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CHAPTER  I 


A 

INTRODUCTION  | 

STATEMENT  OF  THE  PROBLEM  | 

Within  the  Department  of  Defense  (DoD) , military  j 

weapon  system  acquisition  programs  have  had  a history  of  ] 

substantial  cost  growth  in  relationship  to  the  estimated  ! 

program  costs  (23).  A major  factor  in  such  cost  growth  has  t 

t 

been  the  lack  of  reliable  techniques  by  which  a program 

* ! 

manager  can  formulate  cost  estimates  based  both  upon  avail- 

j 

able  information  and  the  risk  and  uncertainty  involved  in 
predicting  future  outcomes  (4;  5;  16;  18;  23).  The  program 
manager  is  the  singularly  selected  individual  chosen  by  the  i 

service  secretary  to  manage  and  control  the  development  of  > 

i 

a major  weapon  system.  Ho  is  responsible  for  the  finan-  | 

i 

cial,  business,  and  administrative  management  of  the 
acquisition  program.  He  is  charged  with  planning,  direc- 
ting, and  controlling  all  phases  of  the  weapon  system 
development,  production,  testing,  distribution,  and  logis- 
tic support.  It  is  the  program  manager  who  is  expected  to 
attain  a smooth,  integrated  effort  within  and  between  these 
phases  (21:45).  However,  without  a reliable  prediction 
technique,  the  program  manager's  ability  to  control  costs 

1 
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is  greatly  impeded.  A realistic  or  accurate  basis  upon 
which  to  assess  a program' s progress  and  to  decide  courses  ^ 
of  action  is  absent.  A tool,  such  as  a feedback  informa** 
tion  system  and  a cost  estimating  technique,  is  needed  by 
the  program  manager  to  understand  and  predict  the  effects 
of  his  decisions  upon  program  cost. 

SIGNIFICANCE  OF  THE  PROBLEM 

Over  the  years,  there  has  been  mounting  criticism 
from  both  private  and  Congressional  sectors  over  the 
increasing  costs  incurred  during  the  military  weapon  system 
acquisition  process.  Such  criticism  has  particularly 
emphasized  the  cost  growth  of  weapon  system  programs  when 
compared  to  the  estimated  program  costs  (1;  2;  21). 

Cost  growth  is  a problem  in  the  weapon  system  acqui- 
sition process  (21:43).  A review  of  pertinent  literature 
indicated  that  the  cost  growth  phenomenon  is  genuine  (1:3; 

2;  4;  5).  A study  conducted  by  Peck  and  Scherer  (16:1) 
during  the  late  1950s  reviewed  the  costs  involved  in  12 
major  weapon  system  acquisition  programs.  The  study 
found  an  average  cost  growth  or  220  percent  and  a standard 
deviation  of  170  percent  (23: 3}  when  comparisons  to  esti- 
mated costs  were  made.  During  tbs  196hs,  a study  conducted 
by  Marschak  (16:1)  showed  an  average  cost  growth  of 
226  percent  for  22  programs.  A study  accomplished  in  1969 
revealed  that  27  out  of  the  35  programs  reviewed,  involved 
coat  growths  which  totaled  $19.9  billion  (23:1). 
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BACKGROUND 


The  phrases , "major  weapon  system"  and  "cost 
growth ,"  are  common  phrases  found  in  Department  of  Defense 
(DoD)  procurement  related  literature.  A definition  of 
these  phrases  should  provide  the  necessary  degree  of  back- 
ground knowledge  to  be  utilized  throughout  this  study. 


The  phrase,  "major  weapon  system,"  originated 
during  the  post  world  War  II  era  (IS) . Prior  to  World  War 
II,  weapons  were  developed  in  a piecemeal  manner.  Increased 
sophistication  and  technological  advancement  led  to  greater 
complexity  in  weapon  systems  resulting  in  increased  product 
cost.  The  phrase,  "major  weapon  system,"  resulted  from  a 
need  to  distinguish  the  more  costly  and  complex  weapon 
systems  from  those  essentially  less  difficult  to  manage. 

DoD  Directive  5000.1,  Major  System  Acquisition , describes 
a major  weapon  system  in  the  context  of  dollar  value, 
national  urgency,  or  recommendations  from  components  within 
the  Department  of  Defense  (32sl). 

The  phrases,  "major  weapon  system"  and  "weapon 
system,"  in  this  study  will  refer  to  . . . 

. . . the  major  items  used  in  national  defense — 
costly  and  technically  complex  items  such  as  planes, 
missiles,  ships,  and  tanks.  [the  phrases  cover]  . . . 
not  only  the  major  item  itself,  including  onboard 
subsystems  such  as  power-plant,  electronics  gear,  and 
armament,  but  also  detached  auxiliary  facilities  and 
equipment  for  such  purposes  as  guidance,  communication, 
supply,  maintenance,  training,  and  data  process- 
ing [16:4]. 
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Weapon  System  Acquisition  Process 

The  process  by  which  weapon  systems  are  acquired  by 
the  DoD  is  termed  the  weapon  system  acquisition  process  (16) 
The  acquisition  process  is  characterized  by  five  distinct/ 
but  "overlapping/"  phases.  The  phases  are  called  the  Con- 
ceptual/ Validation/  Full-Scale  Development,  Production, 
and  Deployment  phases,  and  are  depicted  in  Figure  1 (16;  17) 


Conceptual  Phase.  The  basis  for  a system  program  are 
established  in  the  conceptual  phase.  Program  decisions 
following  this  phase  determine  subsequent  system  design. 

The  technical,  military,  and  economic  basis  for  an  acqui- 
sition program  are  established  through  comprehensive 
system  studies  and  experimental  hardware  development  and 
evaluation.  This  is  a highly  iterative  phase  with  activ- 
ities overlapping  rather  than  occurring  sequentially.  The 
activities  which  usually  occur  are  as  follows: 

1.  Identification  and  definition  of  conceptual 
systems. 

2.  Analyses  (threat,  mission  feasibility,  risk, 
cost,  tradeoff) . 

3.  Design,  experimentation,  and  test  of  opera- 
tional requirements,  key  components,  rx.  tical  subsys- 
tems, and  marginal  technology. 

Completion  of  this  phase  is  marked  by  a meeting  of  the 

Defense  System  Acquisition  Review  Council  (DSARC) . This 

committee  of  key  DoD  officials  reviews  the  program  concept 

for  adequate  support  and  determines  whether  to  proceed  with 

or  terminate  the  weapon  system  development  (16;  17). 
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Validation  Phase.  In  the  validation  phase,  the  major  pro- 
gram characteristics  (technical,  cost,  and  schedule)  are 
refined  and  validated  through  study  and  analysis,  hardware 
development,  or  prototype  testing.  The  quantity  and  level 
of  prototype/hardware  validation  depends  upon  the  nature  of 
the  program,  risks,  and  tradeoffs  that  are  involved,  and 
the  suitability  of  the  system  definition  products. 

Included  in  the  overall  objective  of  the  validation  phase 
are  subsidiary  objectives  to: 

1.  Provide  a basis  for  contract  of  commitments 
prior  to  source  selection. 

2.  Establish  technical  interfaces. 

3.  Define  organization  and  responsibilities. 

4.  Resolve  technical  risk  areas  to  the  extent 
feasible. 

5.  Verify  technical  approaches. 

6.  Establish  schedules  and  cost  estimates  for  the 
full-scale  development  phase. 

7.  Establish  planning  schedules  and  cost  estimates 
for  the  production  phase. 

The  validation  phase  is  generally  conducted  as  a 
DoD-financed  effort  by  two  or  more  contractors  working 
independently  in  collaboration  with  the  using  service  under 
the  system  program  manager's  responsibility.  The  resulting 
design  proposal  is  considered  in  the  Source-Selection  pro- 
cess with  the  objective  of  selecting  the  contractor  and  the 
design  most  likely  to  successfully  complete  the  weapon  sys- 
tem program.  Recommendations  resulting  from  the  Source- 
Selection  process  are  considered  by  the  DSARC  in  light  of 
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cost,  technical  risk,  and  time,  A favorable  decision  by 
the  DSARC  validates  the  need  and  design  of  the  weapon  sys- 
tem and  marks  the  beginning  of  the  full-scale  development 
phase  (16;  17). 


Full-Scale  Development  Phase.  There  are  two  basic  objec- 
tives in  the  full-scale  development  phase.  The  first 
objective  centers  around  design  refinement  in  terms  of 
financial  and  technical  risks  associated  with  long-term 
production.  "Risk"  refers  to  the  possibility  of  more  than 
one  outcome  and  the  objective  and/or  subjective  assignment 
of  a probability  of  occurrence  to  each  outcome  (16:7). 

(The  relationship  of  risk  and  uncertainty  is  discussed 
later  in  more  detail.)  The  second  objective  is  to  fabri- 
cate a working  model  of  the  weapon  system  to  test  system 
performance  characteristics  and  manufacturing  techniques. 
This  effort  is  performed  under  contract  by  the  company 
selected  as  a result  of  the  Source  Selection  process.  A 
DSARC  review  occurs  late  in  this  development  phase  to 
review  progress.  The  decision  to  proceed  with  production 
and  deployment  of  the  weapon  system  is  made  at  the  time  of 
this  DSARC  review  (16;  17). 


Production/Deployment  Phases.  The  last  two  phases  of  the 
acquisition  process  overlap  considerably  in  terms  of  time 
and  activity.  The  production  phase  is  characterized  by  the 
production  of  the  system,  training  equipment,  spares,  and 
facilities  for  operational  use.  Operational  testing  and 
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evaluation  is  conducted  on  early  production  items  to  detect 
and  correct  unacceptable  deficiencies  at  the  earliest 
opportunity.  The  deployment  phase  begins  when  production 
items  are  provided  for  use  to  operational  units.  The 
using  command  begins  operational  tests  on  the  first  unit 
delivered  to  determine  and  improve  operational  capabilities 
of  the  system  and  to  develop  the  most  effective  operational 
tactics,  techniques,  doctrines,  and  standards.  A formal 
transition  during  this  phase  transfers  responsibility  from 
Air  Force  Systems  Command  (AFSC)  to  Air  Force  Logistics 
Command  (AFLC)  for  full  logistics  support  and  management 
of  the  weapon  system  and  related  components  (16;  17). 


Cost  Growth 

The  cost  of  a weapon  system  is  a major  factor  in 
any  decision  to  proceed  with  development  (16:8).  Formal 
weapon  system  cost  estimates  fall  into  three  categories: 
planning  estimates;  development  estimates;  and  current 
estimates.  Planning  estimates  are  preliminary  estimates  of 
total  system  cost  made  early  in  the  conceptual  phase  of 
acquisition.  Development  estimates  are  made  near  the 
completion  of  the  validation  phase  . nd  usually  approach 
the  value  of  the  target  price  of  the  contract  for  develop- 
ment. Current  estimates  are  the  continual  revisions  to 


the  estimated  final  cost  of  the  program  (16).  Cost  growth, 
in  essence,  is  the  positive  difference  between  ultimate, 
actual  cost  and  the  initial  estimates  (1;  4) . 
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The  reasons  why  initial  weapon  system  cost  esti- 
mates, or  estimates  calculated  mid-way  in  the  weapon's 
acquisition  life  cycle  do  not  reflect  the  fullest  cost 
possible — and  therefore,  are  not  good  approximations  of 
later  costs — are  complex.  The  reasons  are  found  in  the 
prevailing  incentive  systems — that  combination  of  rewards, 
conditions,  and  constraints  which  drives  individuals  to 
do  the  things  they  do  (1:2).  The  reasons  are  also  found 
in  the  incentive  systems  implicit  in  Government  procure- 
ment, in  military  command  relationships,  and  in  the 
industrial  marketplace  (1:3). 

Attention  to  cost  growth  has  been  generated  at 
many  levels  of  government  with  differing  degrees  of  concern 
(29).  In  1971,  the  General  Accounting  Office  (GAO)  found 
61  weapon  system  programs  where  costs  were  approximately 
40  percent  over  the  original  estimates  (2:61).  Initially, 
in  order  to  obtain  program  authorizations  and  funding,  the 
program  planning  estimates  had  been  used  by  the  military 
departments  in  their  presentations  to  Congressional  com- 
mittees. The  program  development  estimates,  however,  were 
used  in  subsequent  negotiations  with  contractors  (1:5). 

Of  the  $23.9  billion  difference  that  the  GAO  found  between 
the  developmental  and  current  estimates,  approximately 
$3.2  billion  were  attributable  to  changes  in  the  quantities 
ordered.  In  1972,  the  GAO  found  that  there  was  an  average 
expense  increase  of  31  percent  for  a total  increase  of 
$28  billion  for  77  major  weapon  systems.  To  combat  such 


cost  growths,  the  DoD  has  assigned  and  trained  more  spe- 


cialists to  strengthen  the  government  in-house  capability 


for  making  realistic  cost  estimates  (29:23). 


At  the  same  time  that  cost  growth  has  continued  and 


multiplied,  there  has  been  a trend  for  the  military  forces 


of  the  DoD  to  " . . . cost  less  than  a decade  ago  in  terms 


of  the  people,  goods,  and  services  taken  from  the  economy 


[8:16]."  Using  constant  1974  dollars  (8),  defense  spending 


for  1974  is  $79  billion  as  compared  to  $88  billion  in  1964. 


Looking  further  back,  the  portion  of  the  federal  budget 


allotted  to  defense  spending  has  dropped  from  60  percent  in 


1954  to  approximately  30  percent  in  1974.  However,  while 


defense  spending  has  been  reduced,  total  federal  spending 


has  doubled  (8:10). 


The  question  may  therefore  be  raised,  where  are 


federal  funds  3pent?  Apparently,  the  answer  lies  in  the 


public' 8 demand  for  more  services  other  than  military  pro- 


tection, In  the  past  10  years,  according  to  Crow  (8:10), 


non-Defense  spending  has  rapidly  increased: 


Aid  to  education  has  quintupled;  public  assistance 
tripled;  social  security  tripled;  and  health  care 
increased  from  less  than  one-half  billion  dollars  to 
over  $18  billion — a more  than  forty  fold  increase. 


Since  both  the  military  structure  and  spending  are 


decreasing,  does  this  mean  that  further  reductions  will 


eventually  eliminate  concern  over  the  actual  phenomenon  of 


cost  growth?  In  light  of  the  defense  policy  of  the  United 


States,  the  answer  is  a definite  no.  The  Secretary  of 


Defense,  The  Honorable  Mr.  James  R.  Schlesinger  (25:4), 
indicated  (in  a report  to  Congress)  that  past  experience 
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has  shown  that  the  "...  long  tradition  in  this  country  of 
arming  with  great  haste  when  war  comes  upon  us,  and  dis- 
arming with  even  greater  haste  when  the  war  is  over  . . . ," 
is  no  longer  a viable  basis  for  a military  defense  policy. 
The  costs,  in  terms  of  the  expenditure  of  this  country's 
resources,  would  be  too  great.  Instead,  for  the  foresee- 
able future,  or  until  such  time  as  world  peace  and  security 
can  be  based  upon  something  other  than  a "balance  of  arms," 
the  defense  policy  of  the  United  States  will  be  one  of  main- 
taining "...  a reasonably  stable  level  of  defense  effort 
. . . [25:4]."  According  to  Mr.  Schlesinger  (25:4),  this 
policy  means  that,  at  a minimum,  this  country  must: 

. . , keep  a visible  strategic  nuclear  balance, 
contribute  to  a balance  of  general  purpose  forces  in 
Central  Europe  where  the  bulk  of  Soviet  and  Warsaw  Pact 
forces  are  arrayed  against  NATO,  and  together  with  our 
allies  maintain  a balance  of  naval  forces  to  ensure  the 
freedom  of  the  seas  and  the  protection  of  our  sea  lines 
of  communication  . . . 

However,  to  maintain  a military  force  structure 
within  a perceived  arms  balance  requires  that  the  military 
forces  of  the  United  States  be  modernized  at  a pace  that 
will  keep  this  country  abreast  of  its  potential  enemies 
(25:5).  The  objective  is  to  maintain  a balance  of  arms  in 
the  world  and,  therefore,  relative  peace  and  security.  The 
capability  to  meet  this  objective  is  hindered  by  the  pre- 
sent and  continuing  need  to  stabilize  the  aging  process  of 
major  weapon  systems  and  their  associated  components 
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through  modernization,  which  involves  the  procurement  of 
new  systems  and  the  refinement  of  old  systems.  However, 
the  cost  of  acquiring  new  systems  and  that  of  refining 
existing  capabilities  is  increasing  because  of  inflation 
and  the  complexity  of  the  requirements.  Unless  moderniza- 
tion takes  place,  the  systems  and  components  must  either 
be  retired  without  replacement;  replaced  on  a less  than 
one-to-one  basis  by  more  expensive  systems  and  components; 
or  have  their  service  life  extended,  where  possible.  To 
stabilize  this  aging  process  and  to  provide  an  average  age 
for  each  weapon  system  in  line  with  defense  policy,  recent 
military  studies  (25:6)  have  projected  between  a $1  billion 
and  $2  billion  increase  in  procurement  costs  during  the 
period  of  1980 — 1985  just  to  stop  the  aging  without  chang- 
ing the  level  of  the  military  forces  involved.  This 
estimate  does  not  consider  the  procurement  costs  for  stop- 
ping the  aging  process  of  weapon  system  related  equipment. 
Furthermore,  these  studies  made  optimistic  assumptions  such 
as:  projected  force  structures  will  be  adequate  against 

future  threats;  maintenance  costs  will  be  constant;  and 
systems  acquisition  costs,  in  some  areas,  may  actually 
decrease  (25:7)  . 

In  light  of  the  information  presented  by 
Mr.  Schlesinger  (25),  the  DoD  is  faced  with  a dilemma.  At 
the  same  time  that  costs  are  increasing,  the  DoD  budget  is 
constrained.  The  public’s  demand  for  other  services  from 
the  Government  has  lead  to  an  expansion  of  federal  spending 
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but,  at  the  same  time#  the  capacity  of  available  resources 
to  fulfill  all  needs  is  limited.  What  funding  the  DoD  does 
receive  must  be  used  judiciously.  Since  military  weapon 
systems  must  be  developed  and  procured  within  existing 
budget  constraints,  the  historical  trend  of  cost  growth  for 
weapon  system  programs  inhibits  this  judicial  expenditure 
of  available  funds. 

Cost  growth  needs  to  be  identified  and  curtailed. 
Current  literature  reveals  a number  of  causes  for  cost 
growth  as  shown  in  Figure  2 (23:88-89).  A common  thread 
throughout  the  literature  appears  to  be  the  lack  of  reli- 
able techniques  to  assess  available  information  in  the 
light  of  risk  and  uncertainty  in  predicting  the  future, 
deriving  realistic  cost  estimates,  and  managing  program 
costs.  The  lack  of  reliable  techniques  and  information  is 
presented  and  discussed  by  researchers  and  theorists  and 
is  of  paramount  interest  to  system  program  managers  (5:116, 
124) . 

The  estimates  and  resulting  decisions  are  a function 
of  available  information.  When  all  needed  information  to 
select  an  alternative  is  known,  "...  there  is  no  reason 
for  a wrong  decision  . . . [23:37)."  However,  as  the  num- 
ber of  alternatives  increases,  while  the  available  informa- 
tion ia  incomplete,  the  resulting  estimates  or  decisions 


involve  varying  degrees  of  uncertainty.  The  degree  of 
incompleteness  increases  the  risk  that  a decision  will  be 
wrong  (15:19-32;  23:37-55). 
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In  effect,  the  manager  is  faced  with  estimates  and 
decisions  ranging  in  a continuous  scale  from  certainty  to 
uncertainty.  A degree  of  risk  is  incurred  when  the  avail- 
able information  does  not  provide  complete  certainty.  The 
more  uncertain  the  outcome,  the  greater  the  risk  that  cost 
estimates  will  be  in  error  (16:20-21;  23:37-39).  It  is 
apparent  that  the  earlier  in  the  acquisition  process  that 
a reliable  cost  estimating  technique  can  be  used,  the 
better  the  opportunity  to  control  program  costs  (4:10; 
5:4,123-124;  18:2;  25:7). 

SCOPE  OF  THE  STUDY 

This  research  study  will  replicate  the  Glover-Lenz 
(16)  application  of  the  Martin  Cost  Model  which  incorpo- 
rates uncertainty  and  cost  analysis  addressing  final  weapon 
system  development  cost  (see  Chapter  II) . The  development 
phase  of  the  F-5E,  TIGER  II,  weapon  system  acquisition 
program  was  the  focal  point  of  the  research. 

Within  the  realm  of  full  scale  development,  risk 
and  uncertainty  are  to  be  minimized  prior  to  consideration 
for  production  and  deployment  (17) . This  phase,  therefore, 
was  the  basis  for  a study  of  risk  and  uncertainty  in 
relationship  to  the  final  developmental  cost. 

In  studying  this  relationship,  the  Martin  Cost 
Model  was  used  and  analyzed  as  to  its  capability,  as  a 
mathematical  model,  to  integrate  the  concepts  of  uncer- 
tainty, information,  cost,  and  time  into  a predictive 
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technique  for  accurately  forecasting  the  "expected  final 
cost"  for  the  development  phase  of  a weapon  system  (IS) . 

The  application  of  the  model  was  a retrospective  attempt 
to  thoroughly  consider  significant  information  generated 
by  a recently  completed  development  program. 

Glover  and  Lens  validated  the  Martin  Cost  Model  by 
applying  historical  data  from  the  Short  Range  Attack 
Missile  (SRAM)  development  phase.  Their  conclusion  was 
that s 

The  model  is  a workable  method  to  aid  program 
managers  in  understanding  the  potential  impact  of 
uncertainty  on  the  final  coat  of  a developmental  pro- 
gram [16:42-83]. 

The  present  validation  considered  the  n<r  ..otiatea,  sole 
source  program  of  an  aircraft  weapon  system  addressing 
cost  growth  predictions  for  the  purpose  of  improving  the 
manager's  control  of  program  costs.  A conceptual  depiction 
of  the  procurement  process  is  shown  in  Figure  3 (16) . 

OBJECTIVE  OF  THE  STUDY 

The  primary  objective  of  this  study  was  to  test; 
the  validity  of  the  Martin  Cost  Model  as  a predictor  of 
final  weapon  system  development  cost.  The  cost  model  was 
tested  using  the  F-5E,  TIGER  II,  aircraft  system  develop- 
ment program  as  opposed  to  the  SRAM  missile  system 
development  program  analyzed  in  the  Glover-Lenz  thesis. 
Secondary  purposes  are  the  following: 

1.  Further  develop  the  Martin  Cost  Model  for 
program  manager  use  in  cost  control  functions. 
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2.  Further  develop  measures  of  uncertainty  and 
information  in  weapon  system  acquisition. 


RESEARCH  HYPOTHESIS 


The  research  hypothesis  of  this  report  is  that 
there  is  no  difference  between  the  final  costs  estimated 
by  the  Martin  Cost  Model  and  the  actual  costs  for  a 
completed  weapon  system  development  program. 


i 


CHAPTER  II 


THE  MARTIN  COST  MODEL 


In  Chapter  I,  the  military  weapon  system  acquisition 
programs  of  the  Department  of  Defense  (DoD)  were  described 
as  having  a history  of  substantial  cost  growth  in  relation- 
ship to  the  initial  estimated  program  cost.  The  lack  of 
reliable  techniques  by  which  a program  manager  could  formu- 
late accurate  cost  estimates,  based  upon  available  infor- 
mation, was  described  as  a major  factor  in  such  cost  growth 
phenomenon.  One  recently  suggested  technique  for  providing 
the  accurate  cost  estimate  required  is  the  Martin  Cost 
Model  (16;  23). 

The  Martin  Cost  Model  does  not  address  all  phases 
of  the  DoD  military  weapon  system  acquisition  process. 
Instead,  it  was  created  by  Martin  as  a ".  . . conceptual 
cost  model  . . . (23:33,"  addressing  the  prediction  of  final 
costs  for  the  development  phase  or  subprogram  of  the  acqui- 
sition process.  The  basis  for  the  model  is  mathematical 
and  involves  four  conceptual  parameters:  information, 

uncertainty,  cost,  and  time.  Ir.  predicting  final  costs, 
the  model  predicts  at  the  "macro  level"  of  program  costs 
and  expresses  the  prediction  H.  . . as  a mathematical 
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function  of  uncertainty  relative  to  future  events  in  the 
. . . {16:19]"  development  phase. 

This  chapter  will  present  the  derivation  of  the 
Martin  Cost  Model.  In  the  first  of  three  sections,  the 
basic  assumption  of  the  model,  which  equates  risk  and  uncer- 
tainty, will  be  discussed.  The  second  section  will  present 
the  development  of  the  model  within  the  context  of  the 
information  system  provided  for  the  program  manager  during 
the  decision-making  process.  The  third  section  will  discuss 
the  model  and  its  application. 


RISK  AND  UNCERTAINTY  IN  THE 
DECISION  PROCESS 


The  program  manager  is  part  of  an  on-going  organiza- 
tion system  that  has  as  its  ultimate  objective,  the 
successful  development  of  a weapon  system  within  a given 
budgetary  constraint  (34:160-164).  In  achieving  that 
objective,  cost  estimates  are  used  by  the  program  manager 
in  making  decisions  that  will  affect  the  ultimate  cost  of 
the  program.  As  a decision-maker,  it  is  the  manager’s  task 
to  allocate  "the  available  resources  of  personnel,  money, 

. . . facilities,  i.nd  material  to  the  various  developmental 
steps  . . . [34:163], " within  the  development  phase. 

Certainty 

A key  ingredient  for  such  decision-making  is  infor- 
mation (15:186-187).  If  all  the  information  about  the 
outcomes  of  various  selected  alternatives  is  known  to  the 
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manager#  then  "...  there  is  no  reason  for  a wrong 
decision  . . . 123:37]#"  to  occur.  There  would  be  little 
need  for  data  processing  systems#  experts,  and  other  agen- 
cies to  assist  the  manager  in  the  decision-making  process 
if  all  outcomes  were  known.  Under  complete  certainty  (the 
situation  in  which  only  one  possible  outcome  can  occur  as  a 
result  of  a decision  [16:20]),  the  manager  has  complete 
knowledge.  The  totality  of  a present  situation  and  all 
alternatives  and  resultant  outcomes  for  the  future  are  known. 


Risk 


As  the  situation's  complexity  increases  and  the  num- 
ber of  alternatives  multiply#  the  manager's  knowledge  about 
the  situation  and  the  alternatives  decreases  and  the  infor- 
mation available  to  the  manager  decreases  (14:7;  15:190-191; 
16:19-32;  23:37-55;  34:23,251-267).  The  degree  to  which 
knowledge  and  information  about  the  present  and  the  future 
are  lacking  moves  the  manager  from  the  area  of  certainty 
into  that  of  risk  (the  possibility  that  one  of  several  out- 
comes as  a result  of  a decision  may  actually  occur,  each 
with  a specific  probability  or  relative  frequency  of  occur- 
rence [16:20]).  The  risk  incurred  is  that  a selected  alter- 
native will  not  produce  the  desired  outcome  at  some  future 
time.  A3  that  future  time  becomes  further  removed  from  the 
present,  the  risk  may  also  increase  because  the  basis  for 
foreseeing  the  future  is  incomplete.  Thus,  the  risk  of 
identifying  and  selecting  the  wrong  alternative,  and  thereby 
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making  the  wrong  decision,  is  a function  of  incomplete 
knowledge  and  available  information  as  a result  of  the  tine 
disparity  between  the  present  decision  situation  and  the 
future  outcome  of  the  decision. 


Uncertainty 

In  the  extreme  case,  the  manager  may  have  complete 
knowledge  and  information  concerning  present  events.  He 
does,  however,  lack  virtually  all  knowledge  and  information 
about  the  future  (3:1-89;  4:19-32;  12:9;  14:7,24-25;  15:191; 
16:20;  23:37-55).  At  this  extreme,  the  manager  is  faced 
with  a situation  of  total  uncertainty.  Yet,  in  the  time 
frame  of  the  present,  a decision  may  still  have  to  be  made. 


Conceptual  Relationship  of  Certainty, 

Risk,  Uncertainty 

It  may  appear  that  the  authors  have  made  a distinc- 
tion between  the  concepts  of  ri3k  and  uncertainty.  The 
distinction  would  seem  to  imply  that  the  two  concepts  can- 
not be  equated.  The  program  manager  was  described  a3  a 
decision-maker  who  is  faced  with  making  cost  estimates  to 
assist  in  formulating  decisions  that  will  effect  the  future- 
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ousts  estimated  and  the  decisions  reached  are  a 


function  of  the  manager's  present  knowledge  and  the  infor- 
mation available  to  him  with  which  to  reduce  uncertainty. 

In  effect,  the  manager  is  faced  with  integrating  his  present 
knowledge  U3ing  an  information  system  to  evaluate  the 
probability  of  future  alternatives.  The  difference  between 
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risk  and  uncertainty  is  a matter  of  degree  (15:193;  16:20; 
23:38).  The  concepts  of  certainty,  risk,  uncertainty,  and 
time  may  be  graphically  represented  as  shown  in  Figure  4 
(23 :Ch. 5) . 

If  uncextainty  is  a total  lack  of  knowledge  and 
information  about  the  future,  then,  from  the  standpoint  of 
objective  probability  measurements  (probability  derived  by 
specific  procedures  independent  of  the  problem  (23:43)),  a 
relative  frequency  distribution  for  the  expected  outcomes 
could  not  be  established  (3:9;  11:1-3;  15:193;  23:38-39). 
Certainty  would  be  where  an  event  occurs  with  a probability 
of  one.  As  given  alternatives  are  assigned  a numerical 
probability  other  than  one  or  zero,  the  situation  facing 
the  decision-maker  becomes  more  riuky.  Ri3k  is  the  great- 
est when  the  probabilities  of  alternatives  are  . 5 as  shown 
in  Figure  5 (23:Ch.5).  In  terms  of  objective  probabilities, 
the  probability  assigned  to  an  alternative  is  based  solely 
upon  the  information  or  data  available  about  the  situation. 
No  consideration  is  given  to  the  decision-maker's  prior 
knowledge  and  subjective  probabilities  based  on  degrees  of 
belief  or  intuition  (23:43). 

Frank  Knight  is  credited  with  maintaining  the 
theory  that  if  uncertainty  is  measurable,  then  the  degree 
cf  uncertainty  can  be  assigned  a numerical  probability 
(11:1).  Uncertainty  becomes,  in  effect,  a degree  of  risk. 
If,  however,  uncertainty  cannot  be  measured  and  a numerical 
probability  assigned,  then  real  uncertainty  exists.  The 
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decision-maker  would  appear  to  be  faced  with  a situation  in 
which  the  outcome  and,  thus,  the  decision  as  to  the  correct 
course  of  action,  could  not  be  determined.  This  situation 
would  exist  when: 

. . . the  decision-maker  was  ignorant  of  the  sta- 
tistical frequency  of  events  relevant  to  his  decision; 
a Priori  calculations  were  impossible;  the  relevant 
events  were  in  some  sense  unique;  an  important  once- 
and-for-all  decision  was  concerned  [11:1]. 

The  decision-maker,  however,  is  more  than  a machine 
which  is  fed  information  and  produces  a decision  (34:163). 
The  decision-maker  is  a person,  an  individual.  His  deci- 
sions are  tempered  not  only  by  the  information  that  he 
receives  but,  for  example,  by  his  perceptions  and  back- 
ground or  past  experience  (15:Ch.6-7).  The  information  he 
receives  is  integrated  into  his  existing  knowledge,  based 
upon  perceived  past  experience.  The  cost  estimates  and 
decisions  made  become  a product  of  both  objective  and  sub- 
jective probability  (3:Ch.2;  11;  16:19-40;  23:37-55).  The 
objective  probabilities  are  numerical  values  based  solely 
upon  the  information  that  the  situation  provides  the 
decision-maker.  The  subjective  probabilities  are  con- 
structed from  information  internal  to  the  decision-maker, 
based  on  his  past  experience.  The  objective  and  subjective 
probabilities  are  combined  to  form  one  frequency  distribu- 
tion for  the  alternatives  identified  as  shown  in  Figure  6 
(23 :Ch. 5) . 

One  example  of  interfacing  the  objective  probability 
with  the  subjective  is  presented  by  Daniel  Ellsberg  (11) . 
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Ellsberg  reported  that  subjects  were  presented  with  situa- 
tions involving  varying  alternatives  or  choices  with  vary- 
ing probabilities  of  subsequent  occurrence  and  reward.  In 
each  situation , a choice  had  to  be  made.  Upon  interviewing 
the  subjects , it  was  found  that  the  alternative  or  choice 
selected  in  situations  of  low  or  nonexistent  objective 
probability  of  occurrence  involved  both  the  subject's  know- 
ledge and  the  information  that  he  had  received.  The 
knowledge  or  degrees  of  belief  that  the  individual  held 
were  integrated  with  the  information  presented  to  produce 
a "best  estimate"  of  what  was  likely  to  happen. 

Ellsberg  questions  whether  or  not  the  decision- 
maker or  the  researcher  are  always  able  to  take  qualitative 
data  and  convert  that  data  into  subjective  probability.  At 
least  to  these  authors , there  appears  to  be  sufficient 
techniques  available  with  which  to  integrate  objective  prob- 
ability with  subjective  probability  to  produce  a single 
relative  frequency  distribution  (4) . The  Martin  Cost  Model 
and  the  manipulations  previously  performed  by  Glover  and 
Lenz  (16)  upon  the  Martin  Cost  Model  appear  to  have  vali- 
dated the  existence  of  such  techniques. 

THE  MODEL  WITHIN  AN  INFORMATION  SYSTEM 

As  indicated  in  the  previous  section,  the  program 
manager  is  part  of  an  on-going  organization  system  that 
has  definite  objectives.  Within  the  organizational  context, 
a basic  function  of  the  program  manager  is  in  the.  role  of 
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decision-maker.  As  a decision-maker,  the  program  manager 
is  both  a receptor  Cor  information  from  outside  sources 
and  a source  of  information  himself.  To  the  extent  that 
the  outside  sources  provide  complete  information  about  the 
present  and  future,  the  decision-maker  can  formulate 
alternatives  and  arrive  at  a decision  based  upon  the  objec- 
tive probability  of  each  alternative  actually  occurring  in 
the  future.  However,  to  the  extent  that  outside  sources 
provide  incomplete  information,  the  actual  occurrence  of  a 
given  alternative  in  the  future  becomes  uncertain.  Thus, 
the  future  outcomes  become  grouped  at  the  uncertainty  end 
of  the  certainty-risk-uncertainty  continuum.  With  objec- 
tive uncertainty,  the  decision-making  process  may  appear  to 
be  nothing  more  than  random  selection  of  an  alternative 
upon  which  to  base  a decision.  To  reduce  such  uncertainty, 
the  decision-maker  augments  the  provided  information  with 
his  own  past  experience.  This  augmentation  involves  the 
combining  of  objective  with  subjective  probabilities  based 
upon  his  own  past  experience  and  intuition.  This  combin- 
ing of  the  two  types  of  probabilities  reduces  the  grouping 
of  expected  outcomes  in  the  area  of  uncertainty  and  allows 
the  decision-maker  to  produce  a relative  frequency  distri- 
bution of  expected  outcomes  for  the  alternatives  formulated. 

In  this  section,  the  program  manager,  as  a decision- 
maker, is  viewed  from  both  the  standpoint  of  a receptor  and 
a source  of  information.  This  view  necessitates  a general 
discussion  of  the  immediate  information  system  in  which 
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the  decision-maker  operates.  It  is  on  this  information 
system f with  the  decision-maker  as  both  a receptor  and  a 
source  of  information,  that  the  Martin  Cost  Model  is  based. 

Economic  Uncertainty 

If  the  decision-maker  had  complete  information 
available  as  to  the  potential  outcomes  of  selected  alterna- 
tives, then  the  decision  he  made  would  be  made  with 
complete  certainty.  There  would  be  no  risk  that  a decision 
was  wrong  (23:37).  When  there  is  complete  or  perfect  infor- 
mation, the  situation  corresponds  to  the  concept  of  "pure" 
or  "perfect  competition"  as  defined  by  the  classical 
school  of  economics  (20:106;  22:205;  23:123).  Under 
perfect  competition , the  economic  situation  is  character- 
ized as  (20:106) : 

1.  Containing  markets  with  "...  a large  number 
of  buyers  and  sellers  of  approximately  equal  impor- 
tance . " 


2.  Where  "...  the  products  traded  are  homo- 
geneous * . ."so  that  there  is  no  buyer’s  preference 
as  to  a specific  product. 

3.  Where  the  "...  buyers  and  sellers  always  have 
full  knowledge,  of  the  market." 
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4.  Where  the  "...  buyers  always  act  rationally 
and  (the)  sellers  are  free  to  enter  and  to  leave  the 
market  at  will." 


Each  of  the  four  characteristics  listed  above  must 
be  present  if  perfect  economic  competition  is  to  exist 
(20:106).  Perhaps,  at  one  time,  perfect  competition  did 
exist.  Today,  there  is  little  doubt  that  such  competition 
does  not  exist  (13:34;  20:106).  Therefore,  the  United  States 
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Government , business  firms,  and  industries  are  interacting 
in  an  economic  environment  that  is  less  than  perfect  in 
terms  of  competition  and  hence  the  four  characteristics 
listed  above  do  not  apply  in  to  to. 

This  lack  of  perfect  competition  clearly  appears  to 
be  the  case  in  the  relationship  between  the  DoD  and  the 
DoD  weapon  systems  contractors.  As  Martin  (23:30)  has  indi- 
cated, *.  . , the  larger  number  of  sole- source  negotiated 
procurements  would  seem  to  . . . [characterize  a]  . . . 
bilateral  monopoly  relationship  ..."  between  the  DoD  and 
its  contractors.  Granted,  tt.  . .at  any  given  time,  other 
economic  market  classifications  may  well  apply  . . . [23:30]," 
but  not  under  the  conditions  of  perfect  competition.  In 
terms  of  weapon  system  acquisitions,  such  factors  as  the 
total  dollar  values  involved;  Congressional  budgetary  con- 
straints; changes  in  technology;  advances  in  research  and 
development;  the  political  climate  within  this  country  and 
the  world;  and  the  limited  and  specialized  nature  and  use 
of  the  weapon  systems,  themselves,  tend,  at  a minimum,  to 
place  the  DoD  in  a monopsonistic  position  and  the  contrac- 
tor, as  a member  of  the  "defense  industry",  in  a monopolis- 
tic position  (23:29-36). 

Even  if  the  perfect  competition  characteristics 


labeled  on  the  previous  page  as  one,  two,  and  four  were 


present,  the  lack  of  characteristic  number  three,  full 


knowledge,  would  prohibit  perfect  economic  competition. 
Instead,  the  lack  of  full  knowledge  would  create  economic 
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uncertainty  (16:32).  This  uncertainty  appears  evident  in 
the  historical  trend  of  cost  growth  where  the  actual , final 
costs  of  weapon  system  acquisition  programs  have  deviated 
widely  from  the  original  cost  estimates  (refer  to  Chapter  I) . 
Furthermore,  such  factors,  as  previously  mentioned,  not 
only  contribute  to  the  general  absence  of  perfect  compe- 
tition characteristics  one,  two,  and  four,  but  also  con- 
tribute to  internal  and  external  economic  uncertainties  for 
both  the  buyer  and  seller  (23:29-36).  Internal  uncertain- 
ties, such  as  those  found  in  ".  . . technical  design,  . . . 
[physical]  component  integration,  and  financial  stabil- 
ity ..."  and  the  external  uncertainties,  such  as  those 
pertaining  to  requirements  or  demands  for  the  procurement 
of  a given  weapon  system  and/or  components,  may  be  gener- 
ated (23:34-35).  Possible  areas  of  uncertainty  are  further 
exemplified  in  Table  1 (23:41-43)  which  presents  uncertainty 
as  being  grouped  under  four  taxonomic  classes. 

Since  the  DoD,  in  relationship  to  its  weapon  sys- 
tem contractors,  is  interacting  in  an  economic  environment 
of  less  than  perfect  competition,  there  is  an  element  of 
uncertainty  involved  in  the  decision-making  process  in 
which  the  program  manager  functions.  Therefore,  sources 
of  information  from  outside  the  program  manager,  himself, 
may  not  be  sufficient  for  the  formulation  of  alternatives 
and  the  assignment  of  objective  probabilities  to  future 
outcomes.  Instead,  the  program  manager  may  have  to  rely 
upon  an  integration  of  outside  sources  of  information  with 
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Table  1 

Uncertainty  Taxonomy 


Description 


Comment 


Environmental : 

la.  Nature  la. 


b.  Social  and  b. 

Political 


c.  Communication  c. 

Media 


d.  Time  d. 

2a.  External  2a. 

b.  Internal  b. 

3a.  Exogenous  3a. 

b.  Endogenous  b. 


The  uncertainty  is  related  to 
natural  factors#  such  as 
storms  and  floods. 

The  term  relates  to  the  impos- 
sibility of  being  able  to  pre- 
dict with  any  precision  the 
actions  of  social  and  politi- 
cal groups. 

The  disparities  that,  exist  in 
the  access  which  people  have 
to  the  various  informational 
media.  The  differences  result 
in  ignorance  on  the  part  of 
many  groups  and  individuals. 

The  passage  of  time  results  in 
changes  which  can  distort  the 
results  of  decisions  based  on 
a past  3tate-of-af fairs. 

These  uncertainties  relate  to 
factors  external  to  a project 
which  can  impinge  on  final 
resuits. 

Internal  uncertainties  comprise 
those  stemming  from  the  techni- 
cal approach  taken,  etc. 

The  stimulus#  initiating  a 
given  change,  comes  from  out- 
side the  organization. 

The  stimulus,  initiating  the 
change  originates  within  the 
organization. 
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Table  1 (continued) 


OP 


i 


Description 

Functional : 
la.  Business 


b.  Financial  Risk 


c.  Technological 
Uncertainty 


d.  Production 


Informational : 

la.  Anticipated 
Unknowns 


b.  Unanticipated 
Unknowns 

2a.  Known  Unknowns 


Comment 


la.  The  firm  is  uncertain  about 
its  future  income  stream. 

The  risk  is  associated  with 
the  firm's  operation. 

b.  The  uncertainty  is  generated 
by  the  ratio  of  debt  to  equity 
in  the  capital  structure.  The 
amount  of  earnings  available 
to  common  stockholders.  For 
contracting  the  risk  of  profit 
or  loss  on  an  individual  con- 
tract is  involved. 

c.  Changes  in  the  state-of-the- 
art  can  render  a weapon  obso- 
lete. Thus,  uncertainty  exists 
as  to  how  long  a weapon  can 
remain  in  the  operational 
inventory. 

d.  Most  products  represent  an 
integration  of  component  parts. 
Should  a part  not  be  avail- 
able, then  the  finished  product 
cannot  be  ready  on  time  and 
even  its  cost  can  be  affected. 


la.  The  unknowns  in  this  class  are 
those  that  a contractor  is 
aware  of.  The  problem  area 
is  anticipated, 

b.  These  unknowns  cannot  be  fore- 
seen. 

2a.  The  facts  the  contractor  knows 
that  he  does  not  know. 


b.  Unknown  Unknowns 


b.  The  unknowns  the  contractor 
does  not  anticipate. 
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Table  1 (continued) 


Description  Comment 


Technical : 

la.  Uncertainty  la.  The  known  is  completely  domi- 

nated by  the  unknown.  The 
probability  distributions  for 
future  events  are  not  known. 

b.  Risk  b.  A decision  leads  to  one  of  a 

specific  number  of  well 
defined  alternatives.  The 
totality  of  outcomes  for  a 
given  variable  can  be 
described  by  a probability 
distribution. 


c.  Certainty  c.  Each  decision  leads  to  « pre- 

dictable outcome.  No  doubt 
as  to  the  final  outcome  is 
possible. 

2a.  Subjective  2a.  The  term  relates  to  the  proba- 

bilities assigned  to  ar.  event 
and  which  are  wholly  based 
on  the  observation  choice. 

b.  Objective  b.  These  probabilities  are 

derived  by  specific  proce- 
dures independent  of  the 
problem  being  confronted. 
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his  past  experience  and  intuition  to  arrive  at  alterna- 
tives, assign  probabilities  to  each  alternative's  future 
outcome,  and  render  a decision. 

The  interaction  of  the  program  manager  as  a decision" 
maker  and  as  a receptor  of  information  from  outside  sources 
implies  that  there  exists  some  type  of  information  system 
in  which  the  manager  functions.  This  information  system 
shall  be  discussed  next  followed  by  a discussion  of  the 
Martin  Cost  Model. 


Information  System 

As  previously  indicated,  a basic  function  of  the 
program  manager  is  that  of  a decision-maker  within  an 
on-going  organization  system.  Part  of  that  organization 
system  is  the  information  subsystem  ir.  which  the  program 
manager  functions  as  a decision-maker  (23:118-120). 

The  information  subsystem  of  the  program  manager 
is  depicted  in  Figur«  7 (23:119).  The  subsystem  "... 
embodies  features  of  Shannon's  communication  system  and 
closedloop  feedback  mechanism  of  Wiener's  cybernetic  theory 
(16:31]."  Although  the  subsystem  portrayed  may  seem  sim- 
plistic in  nature,  the  subsystem,  as  a model  of  the 
decision-maker's  environment,  will  suffice  for  developing 
the  underlying  concepts  and  issues  of  the  Martin  Cost  Model 
(16:Ch.II;  23:Ch.IX;  28:33-35). 

The  model  in  Figure  7 is  a closed  system  relative 
to  the  decision-maker  and  his  immediate  environment  (23:118) 
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In  viewing  the  model,  emphasis  should  be  placed  upon  the 
receipt  of  information  by  the  decision-maker  rather  than 
upon  the  transmission  of  information  (23:119).  Under  such 
a view#  information  is  considered  an  input  into  the  infor- 
mation system  for  the  purpose  of  giving  the  decision-maker 
. feedback  about  the  state  of  [his]  . . . [environment] 

. . . and  the  financial  status  of  expenditures  under  a 
development  contract  [23:47]."  Such  information  is,  there- 
fore, germane  to  the  developmental  phase  of  a given  weapon 
system  acquisition  program  and  forms  the  basis  upon  which 
cost  estimates  for  the  phase  are  to  be  derived.  The  infor- 
mation can  be  considered  as  a "commodity"  (23:47)  which 
"...  add[s]  to  a representation  of  what  is  known,  believed, 
or  alleged  to  be  so  [34:22]."  Thus,  information  has  the 
potential  of  increasing  knowledge  (34:22).  The  term 
information  can  be  further  delimited  by  the  concepts  of 
amount  and  usefulness.  These  two  concepts  will  be  explored 
in  subsequent  paragraphs.  The  immediate  point,  however,  is 
that  for  the  purpose  of  the  Martin  Cost  Model  it  is  not 
necessary  to  expound  upon  a strict,  detailed  definition  of 
what  information  is  (23:45).  The  primary  concern  is  the 
amount  and  usefulness  of  the  input  information  reqardless 
of  the  form  of  that  information  or  the  means  by  which  that 
information  is  transmitted. 


If  thr*  purpose  of  the  information  subsystem  is  to 
provide  the  decision-maker  with  information  in  the  form  of 
feedback,  then  a logical  place  from  which  to  initiate 
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further  discussion  of  the  model  is  the  activity  node  in 
Figure  7 (16:Ch.II;  23:Ch.IX;  28:33-35).  In  the  most 
general  of  terms » the  activity  node  represents  the  entire 
environment  of  the  developsient  phase  external  tc  the 
decision-maker.  The  activity  node  is  the  outside  source  of 
information  for  the  decision-maker.  This  node  includes, 
and  is  focused  upon,  the  activities  of  both  the  contractors, 
who  are  working  upon  the  development  of  a weapon  system, 
and  the  administration  of  the  development  contracts  (16:32; 
23:119-120). 

As  time  passes,  the  activity  node  generates  infor- 
mation about  the  implementation  and  progress  of  the  develop- 
ment phase.  This  information  is  gathered  and,  to  varying 
degrees,  collated  by  the  environmental  sensors  node.  The 
sensors  include  both  civilian  and  DoD  personnel  and  the 
various  reporting  subsystems  used  in  data  collection.  These 
sensors  also  include  the  manipulation  of  the  information 
collected  into  forma  that  are  acceptable  for  computer 
processing  (28). 

Once  the  information  has  been  collected  and  proc- 
essed through  the  computer  node,  the  decision-maker  has 
information  products  that  provide  feedback  to  him.  Feed- 
back not  only  indicates  the  current  and/or  projected  state 
of  the  development  phase,  but  also  gives  an  indication  of 
how  his  past  decisions  have  affected  the  progress  of  activ- 
ities. This  feedback,  in  turn,  forms  the  basis  for  formu- 


lating new  alternatives  and  the  modification  of  prior  and/or 
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the  generation  of  new  decisions.  These  subsequent  deci- 
sions are  then  passed  on  to  the  activity  node  to  further 
circumscribe  the  content  of  the  activities  to  be  performed. 

Noise.  Discussion  about  the  model  is  now  complete  except 
for  one  node,  that  of  noise  sources.  Within  communication 
theory,  noise  refers  to  "distortion,"  "error,"  or  "extran- 
eous material"  being  added  to  the  message  (28:19;  34:26). 
Noise  could  result  from  such  factors  as  ".  . . cursory  cost 
analysis,  contractor  underpricing,  extraneous  design 
requirements,  technical  changes,  . . . inadequate  task 
definition  . . . [23:127-128],"  or  errors  and  distortion 
due  to  incorrect  keypunching  and  reporting.  The  message 
which  is  being  transmitted  from  the  activity  node  to  the 
environmental  sensors  is  a ".  . . set  of  signs  . . . [or 
symbols]  . . . that  are  intended  to  convey  information  . . 
[34:24]"  to  a receptor  which,  in  this  case,  is  ultimately 
the  decision-maker.  In  effect,  during  the  transmittal  and 
receipt  processes,  noise  may  be  introduced  (28:34).  This 
added  information,  in  the  form  of  distortion,  error,  or 
extraneous  material,  obscures,  to  a certain  degree,  the 
original  information  (34:26).  This  noise  and  the  original 
information  that  the  message  was  to  convey  cannot  be  com- 
pletely separated.  Therefore,  the  final  receptor,  i.e., 
the  decision-maker,  is  faced  with  greater  latitude  in  inter 
preting  the  content  of  the  message.  As  shall  shortly  be 
seen,  this  latitude  in  interpretation  tends  to  increase 
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the  decision-maker's  uncertainty  as  to  the  exact  meaning  of 
the  information  provided  in  the  message. 

To  expand  upon  the  concept  of  noise  and  to  properly 
relate  this  concept  to  the  decision-making  process  within 
an  information  system , requires  a further  explanation  of 
what  is  meant  by  the  term  information.  Based  upon  develop- 
ment of  Shannon's  presentation  (16:Ch.II;  23:Ch.IX;  28:8-16, 
100-103) , information  can  bo  considered  to  be  the  number 
(amount)  of  alternatives  or  choices  that  are  available  to 
the  receptor  for  selecting  and  interpreting  a message. 

Since  each  interpretation  would  convey  information  that  was 
somewhat  different  from  what  the  original  message  intended, 
the  receptor  is,  in  effect,  confronted  with  a number  of 
messages,  each  with  a meaning  that  varies  from  the  other. 
Thus,  the  extent  to  which  noise  affects  the  message  origi- 
nating at  the  activity  node,  the  amount  of  information,  or 
the  number  of  alternatives  or  outcomes  presented  to  the 
receptor  are  compounded.  However,  the  issue  is  not  the 
meaning  of  the  information  contained  in  each  message,  but 
rather  the  amount  of  information  presented  as  represented 
by  the  totality  of  the  situation  confronting  the  receptor. 

T.iformation,  in  terms  of  amount  rather  than  meaning, 
is  exemplified  by  the  following  situation.  Consider  a 
simplistic  situation  in  which  the  activity  node  has  gener- 
ated one  message.  This  message  is  affected  by  noise  prior 
to  being  processed  by  the  environmental  sensors.  As  a 
result  of  noise,  the  decision-maker  (the  ultimate  receptor) 
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is  presented  with  two  "alternative  messages"  from  which  to 
choose.  Somewhat  the  same  situation  was  presented  by 
Shannon  (26:9)  who  went  on  to  elaborate  as  follows: 

To  be  somewhat  more  definite,  the  amount  of  infor- 
nation  is  defined,  in  the  simplest  cases,  to  be  measured 
by  the  logarithm  of  the  number  of  available  choices. 

It  being  convenient  to  use  logarithms  to  the  base  2, 
rather  than  common  or  Briggs'  logarithm  to  the  base  10, 
the  information,  when  there  are  only  two  choices,  is 
proportional  to  the  logarithm  of  2 to  the  base  2.  But 
this  is  unity;  so  that  a two-choice  situation  is  charac- 
terized by  information  of  unity,  ....  This  unit  of 
information  is  called  a "bit,"  this  word,  first  sug- 
gested by  John  W.  Tukey,  being  a condensation  of 
"binary  digit."  When  numbers  are  expressed  in  the 
binary  system  there  are  only  two  digits,  namely  0 and  1, 
just  as  ten  digits,  0 to  9 inclusive,  are  used  in  the 
decimal  number  system  which  employs  10  as  a base.  Zero 
and  one  may  be  taken  symbolically  to  represent  any  two 
choices,  as  noted  above;  so  the  "binary  digit"  or  "bit" 
is  naturally  to  associate  with  the  two-choice  situation 
which  has  unit  information. 

Consider  a situation  in  which  the  receptor  is  con- 
fronted with  32  alternative  messages  (28:9-10).  The  choice 
as  to  which  message  the  receptor  should  select  is  equally 
likely.  Using  logarithms  to  the  base  2,  the  situation 
contains  a total  amounting  to  five  bits  of  information. 

The  five  bits  were  derived  from  the  calculation  log2  32  « 5 
since  2®  » 32. 


Entropy.  To  further  measure  the  amount  of  information  pre- 
sented to  the  receptor  or  the  amount  of  information  within 
the  system  over  time,  both  Martin  and  communication  theory 
have  borrowed  the  concept  of  entropy  from  the  physical 
sciences  (16:21-30;  23:121-123;  28:12-16,19-22,100-103; 
34:23,257-265).  The  concept  of  entropy  was  derived  from 
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thermodynamics.  Entropy  is  a measure  of  the  "disorder"  or 
the  "...  improbability  of  the  next  state  . . . [34:21]" 
of  a closed  system  (23:121).  What  is  measured  is  ".  . . the 
degree  of  randomness  ..."  within  the  situation  (28:12). 
Over  timer  in  accordance  with  the  second  law  of  thermody- 
namics # the  tendency  is  for  entropy  to  increase.  Therefore , 
the  system  presents  an  increasing  amount  of  disorder  or 
lack  of  organization  with  respect  to  the  amount  of  informa- 
tion provided  the  decision-maker.  As  the  number  of  messages 
confronting  the  decision-maker  increases , the  probability 
of  selecting  one  message  over  another  as  the  alternative 
upon  which  to  base  a decision  decreases.  The  situation 
eventually  may  become  one  in  which  all  alternatives  appear 
to  the  decision-maker  as  equally  likely  in  probable  out- 
come. Such  a situation  is  one  of  complete  uncertainty  as 
to  the  future.  The  relationship  between  entropy  and  uncer- 
tainty is  direct  (23:122). 


Measurement  of  Entroi 


The  measurement  of  the  amount  of  information  pre- 


sented to  the  decision-maker  is  based  upon  calculating 
entropy  as  a . . function  of  probability  . . . [16:26]." 
Ac  described  by  Glover  and  Lenz  (16:26),  the  term  proba- 
bility: 


. . . refers  to  the  relative  frequency  of  occurrence 
of  an  outcome  in  a series  of  trials.  [In  subsequent 
examples  of  entropy  calculations,  it  shall  be  emphasized 
that  the  probabilities  referred  to  are  based  upon  the 
decision-maker ' s use  of  objective  and/or  subjective 
probability  techniques  applied  to  the  alternatives 
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presented  (16:2?).]  In  the  Martin  Cost  Model,  the 
occurrence  of  a given  cost  is  considered  as  [an]  out- 
come. Each  alternative  choice  available  to  the 
d&cision-maker  may  be  expressed  as  a dollar-cost  out- 
come. The  measure  of  the  amount  of  information , 
entropy/  is  a function  of  the  probability  of  a given 
outcome/  or  in  the  program  system/  [i.e.,  development 
phase] / the  cost  outcome  prescribed  by  a choice  of  an 
alternative.  Where  the  number  of  alternatives 
increased/  the  measure  of  entropy  increased. 

In  calculating  entropy/  theoretically,  the  messages 
presented  to  the  decision-maker  might  be  considered  as 
31 . . . continuously  variable  functions  of  time  . . . [which] 
can  assume  an  infinite  number  of  bits  for  exact  specifica- 
tions [34: 260]. " Under  such  a consideration,  entropy,  which 
is  presented  by  the  symbol  H,  would  be  expressed  as: 


H * - 


+«• 

f p(x)log  p(x)dx 


which  represents  "...  the  entropy  of  a continuous  dis- 
tribution with  the  density  distribution  in  a function  space 


p(x)  . . . [34:260]." 


However,  in  "practical  situations,"  messages  cannot 
".  . . require  an  infinite  frequency  band  for  adequate 
transmission  and  interpretation  [34:260]."  Therefore, 
entropy,  H,  calculations  can  be  reduced  to  a discrete  set 
of  probabilities  and  expressed  as: 


H 


log  p. 


i»l 


where  p^  represents  the  probability  of  a given  outcome,  n 
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is  the  total  number  of  outcomes  presented , and  K is  a posi- 
tive constant  and  scaling  factor  (28:87;  34:257). 

Since  K in  the  above  expression  is  a scaling  factor 
affecting  only  the  absolute  value  of  H,  K can  be  deleted 
(16:26;  26:167-168).  This  deletion  reduces  the  entropy 
calculation  to: 

n 
H 

i«l 

In  the  above  mathematical  expressions,  entropy,  H, 
reflects  the  total  amount  of  uncertainty  present  and  avail- 
able in  the  information  system.  This  total  amount  reflects 
the  uncertainty  confronting  the  decision-maker  in  the 
decision-making  process.  Uncertainty,  i.e.,  entropy, 
reaches  its  maximum  when  the  probability  of  each  alterna- 
tive's outcome  is  equally  likely  (28:15).  The  probability 
distribution  for  the  possible  outcomes  is  flat  (26:171). 
Where  the  probability  distribution  is  not  flat,  the  proba- 
bility of  each  outcome  occurring  is  not  equal.  As  the 
inequality  of  the  probabilities  increases,  the  outcomes 
spread  out  and  away  from  the  central  value  (26:170).  Thus, 
entropy  can  be  further  viewed  and  interpreted  as  a measure 
of  how  flat  the  probability  distribution  is  and  represents, 
in  a single  value,  H,  the  total  amount  of  uncertainty  con- 
tained in  the  system  (26:170).  As  such,  it  also  reflects 
the  variance  "...  about  . . . [the]  . . . most  probable 
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outcome  [16:26].”  If  all  outcome*  were  equally  likely, 
then  disorder#  as  defined  by  entropy#  is  complete. 

The  mathematical  expression  for  H can  be  transformed 
into  the  following  ratio  (16:29;  28:13;  34:258): 

n 


-E 

i«0 

at  — a.  ,-V 


Pilog  pt 
Hmax 


and 


Hma; 


log  n 


where  HTUY  represents  maximum  entropy  and  h represents  the 
relative  entropy  of  the  system.  At  unity  or  complete  dis- 
order# h«l . For  no  disorder  (complete  order  or  certainty) # 
h»0.  Therefore#  the  range  of  values  for  h can  be  expressed 
as:  O&h&l.  For  example , if  h were  calculated  and  found  to 

be  .60#  then  the  relative  entropy  in  the  system  is  about 
60  percent  (28:13). 


Negentropy 

From  a logical  standpoint#  if  entropy  reflects  the 
disorder  contained  in  the  system#  then  the  opposite  of 
entropy  or  disorder  would  be  order.  The  degree  of  order  in 
the  system  can  be  expressed  as  (16:30): 

IE  - 1 - h 

where  1 represents  unity  within  the  system#  h represents 
relative  entropy#  and  IE  represents  order.  Order  in  commu- 
nications theory  is  referred  to  as  negentropy.  If  h were 
.60  as  in  the  above  example#  then  negentropy  would  be  .40. 
In  this  situation#  there  is  40  percent  as  much  information 
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as  would  exist  under  certainty.  Together,  relative  entropy 
and  negentropy  must  equal  system  unity  or  1,  i.e. , 

IE  + h * 1 in  the  above  expression  (16:30) . 

The  amount  of  negentropy,  IE,  in  the  system, 
represents  more  than  just  order  (23:120-123).  To  the 
extent  that  there  is  order,  there  is  certainty.  As  the 
amount  of  order  in  a system  increases,  the  probability 
distribution  for  the  outcomes  loses  its  flatness.  The 
probabilities  are  spread  out  and  are  less  bunched  or  grouped 
together.  The  probability  for  each  outcome  becomes  increas- 
ingly distinctive  in  comparison  to  the  other  outcomes. 
Increased  order  within  the  system  allows  the  decision- 
maker <.3  discern  differences  between  the  alternative  out- 
comes. The  differences  are  in  terms  of  the  probability  of 
future  occurrence  for  each  outcome.  The  uncertainty  in 
selecting  an  outcome  upon  which  to  base  a decision  is 
reduced.  Certainty  is  increased  and  the  risk  of  a wrong 
decision  is  lowered.  The  information  contained  in  the 
messages  becomes  more  meaningful  and,  thus,  more  useful  in 
making  a decision.  Therefore,  the  degree  of  negentropy 
provided  in  the  system  is  not  only  a measure  of  the  order 
within  the  system,  but  also  a measure  of  information  content 
with  respect  to  the  meaning  and  usefulness  that  the  messages 
provide  in  reaching  a decision. 
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Informational  Efficacy.  To  emphasize  that  negentropy  not 
only  represents  the  order  or  certainty  within  the  system. 
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but  also  the  degree  of  meaning  and  usefulness  the  messages 
provide«  Martin  denotes  negentrcpy  as  IE  (23:120-121).  IE 
refers  to  the  informational  efficacy  of  the  system  and 
encompasses  the  total  content  (order/certainty  and  meaning/ 
usefulness)  of  the  messages  provided  as  measured  by  negen- 
tropy . 

Example  of  Calculation.  Consider  the  following  example  of 
entropy  and  informational  efficacy  calculations  (23:124). 
The  information  subsystem  has  provided  the  program  manager 
with  five  alternatives.  Each  alternative  is  a cost  esti- 
mate of  program  costs.  Since  the  economic  environment  is 
characterized  by  imperfect  competition!  there  may  be  a 
degree  of  uncertainty  associated  with  the  cost  estimates. 

In  this  particular  situation,  there  is  uncertainty  and  the 
program  manager  has  had  to  use  both  objective  and  subjec- 
tive probability  techniques  to  assign  a probability  of 
future  occurrence  to  each  alternative  outcome  or  cost  esti- 
mate. The  cost  estimates  and  their  probabilities  are  shown 
in  Table  2a. 

Given  the  five  alternatives  and  their  associated 
cost  estimates  and  probabilities,  an  expected  or  most 
probable  cost  in  implementing  the  program  activity  can  be 
calculated.  I^ble  2a  can  now  be  expanded  to  reflect  the 
expected  value  as  shown  in  Table  2b.  The  expected  value 
is  not  a cost  estimate  taken  directly  from  the  information 
provided  to  the  program  manager.  Instead,  the  expected 
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value  or  cost  is  a weighted  average  of  what  the  program 
coats  should  be  in  light  of  the  uncertainty  that  a given 
outccme  will  actually  occur  (26:28-29). 

The  entropy  or  degree  of  uncertainty  associated 
tfith  the  above  probability  distribution  of  cost  estimates 
is  represented  in  Table  3.  The  relative  entropy  and 
informational  efficacy  are  presented  in  Table  4. 

As  can  be  seen  from  Table  4,  the  situation  con- 
fronting the  program  manager  is  extremely  uncertain 
(li*.9843).  The  alternatives  presented  are  of  little  meaning 
or  use  (IE*. 0157)  in  making  the  decision.  As  time  passes* 
the  final  costs  actually  involved  may  deviate  greatly  from 
the  expected  cost  (525  (see  Table  2b] ) or  best  estimate  of 
what  the  costs  should  be  in  light  of  the  uncertainty 
involved.  From  the  background  provided  in  Chapter  I*  the 
form  that  the  deviation  would  be  expected  to  take  is  in 
terms  of  a significant  cost  growth. 

Table  5 depicts  a situation  in  which  entropy  is 
lower  and  the  informational  efficacy  is  higher  than  the 
previous  example. 

In  comparing  the  values  presented  in  Table  5 with 
those  presented  in  Tables  3 and  4*  it  can  be  seen  that  the 
maximum  entropy  has  remained  the  same  since  there  are  still 
five  alternatives  confronting  the  program  manager.  However, 
Table  5 reflects  a decrease  in  the  entropy  and  relative 
entropy  values  by  approximately  .28  while  informational 
efficacy  has  increased  by  approximately  .28  in  relationship 
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to  system  unity  or  one.  Furthermore # the  probabilities 
presented  in  Table  5 are  different  from  those  presented  in 
Table  3.  As  shown  in  Figure  8,  the  probabilities  of  the 
alternatives  presented  in  Table  5 have  pronounced  peaks  or 
are  less  flat  than  the  probabilities  in  Table  3.  In  Figure 
8#  the  individual  probabilities  are  less  bunched  or  grouped 
together.  Both  Table  5 and  Figure  8 reflect  that  alterna- 
tives number  3 and  4 appear  to  be  much  more  likely  to  occur 
in  the  future  than  do  the  other  alternatives.  What  has 
happened  is  that  the  informational  efficacy  of  the  informa- 
tion subsystem  has  been  increased#  thereby  presenting  more 
meaningful  or  useful  information  to  the  program  manager. 

This  increased  meaning  or  usefulness  of  the  information 
provided  has  allowed  the  program  manager  a greater  capability 
in  distinguishing  between  alternative  outcomes.  The  greater 
capability  is  reflected  in  the  probabilities  assigned  and 
in  the  measurement  of  informational  efficacy  as  presented 
in  Table  5. 

The  Martin  Cost  Model 

In  Chapter  I#  it  was  indicated  that  more  accurate 
program  cost  estimates  are  required  if  available  resources 
are  to  be  used  judiciously  and  the  cost  growth  phenomenon 
curtailed.  However#  as  already  noted#  the  weapon  system 
acquisition  programs  exist  in  an  economic  environment  of 
imperfect  competition.  Various  factors  operating  within 
that  environment  may  render  the  situation  for  a given 
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development  program  highly  uncertain  in  terms  of  final 
costs.  It  is  possible  that  informational  efficacy  could  be 
increased  to  the  point  where  the  costs  were  accurate  within 
a given  situation.  The  funds  required#  however#  may  make 
the  attempt  at  increasing  informational  efficacy  to  the 
needed  level  prohibitive  (23:130-1321.  Furthermore#  since 
there  is  imperfect  competition#  the  degree  of  informational 
efficacy  required  to  provide  sufficient  certainty  for 
accurate  cost  estimates  may  not  be  physically  obtainable 
from  the  environment.  Therefore#  at  best#  it  would  appear 
to  the  authors  of  this  study  that  the  discussion  presented 
so  far#  if  implemented  in  a development  program#  would  only 
provide  the  program  manager  with  the  following: 

1.  A measure  of  the  degree  of  entropy  and  informa- 
tional efficacy  or  uncertainty/certainty  involved  in 
the  situation. 

2.  A probability  distribution  of  alternative  cost 
outcomes  and  an  expected  value  which  represent  less 
useful  and  meaningful  information  as  entropy  increases. 

To  the  degree  that  entropy  exists#  the  program  manager  is 
still  left  in  a position  of  not  knowing  what  the  final  pro- 
gram costs  will  be  until  the  program  is  completed.  Based 
upon  the  cost  estimates  generated#  which  may  have  very  low 
probabilities  cf  actual  occurrence#  how  far  should  the 
actual  costs  incurred  rise  above  the  estimates  before  possi 
ble  cost  growths  attributable  to  less  than  the  judicial  use 
of  resources  are  indicated  and  corrective  management 
actions  are  taken  to  reduce  such  growths? 
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In  an  attempt  to  provide  a management  tool  with 
which  to  curtail  and  control  coat  growth  within  a develop- 
ment program,  Martin  has  formulated  a mathematical  model 
which  integrates  the  concepts  of  uncertainty,  informational 
efficacy,  costs,  and  time  into  a predicative  technique  not 
only  concerned  with  initial  cost  estimates  but  with  the 
expected  final  costs  of  the  program  (23:125-130).  The  basis 
for  the  prediction  of  the  final  expected  cost  is  the  cost 
estimate  or  "target  price"  agreed  upon  by  the  DoD  and  the 
DoD  contractor  in  relationship  to  the  degree  of  informa- 
tional efficacy  involved  in  arriving  at  that  estimate  or 
price  (16:34-35).  The  model  is  mathematically  expressed 
as  follows  (16:35;  23:126): 

^*1 

* — ...  ■■■■«, 

E ie 

Within  this  expression,  C£  represents  the  expected  final 
cost  of  the  program.  The  cost  referred  to  is  an  economic 
cost  which  "...  includes  both  the  . . . [contractor’s] 

. . . cost  and  profit  [16:32]"  under  DoD  contract.  Cost, 
therefore,  refers  to  the  total  amount  to  be  paid  to  the 
contractor  by  the  DoD  (16:32).  Cj  represents  an  initial 
cost  estimate  or  target  price,  and  also  refers  to  the  total 
amount  to  be  paid  (16:32).  IE  represents  the  informational 
efficacy  involved  in  the  generation  of  Cj.. 

For  example,  consider  again  the  situation  presented 
in  Table  5.  Suppose  that  the  target  price  finally  agreed 
upon  by  the  DoD  and  the  contractor  was  alternative  number  3 
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with  a cost  of  385  at  a probability  outcome  of  .55.  Using 

the  Martin  Cost  Model,  would  be  385  and  IE  would  be 

.2918.  C would  thus  be  calculated  to  be  approximately 
£ 

1319.4.  In  this  case,  C_  represents  approximately  a 243 

Ci 

percent  increase  over  the  initial  target  price.  This 
increase  may  suggest  to  the  program  manager  that  more  infor- 
mation should  be  acquired,  potential  cost  growth  factors 
identified,  and  action  taken  to  preclude  this  possible  243 
percent  cost  increase. 

The  above  situation  is  projected  through  time  in 
Figure  9 (16;37-39).  The  program  is  initiated  at  time  Tq 
with  the  awarding  of  the  program  contract.  Time  will  pass 
in  intervals  of  T,  such  as  fiscal  quarters  per  year,  until 
the  contract  is  terminated  and  the  program  completed  in  the 
nth  period  or  Tn.  At  time  TQ,  the  program  manager  plotted 
the  expected  costs  of  the  program  over  time  based  on  the 
expected  final  cost,  C£,  of  1319.4.  To  plot  this  line, 
labeled  C£,  it  is  assumed  that  there  are  zero  start  up  costs 
at  time  TQ  due  to  the  contractor  already  belonging  to  the 
defense  industry  (16:36).  It  was  further  assumed  that  coats 
would  accumulate  over  time  linearly  (16:36).  When  period 
Tn  becomes  the  present  rather  than  the  future,  the  actual, 
final  costs  will  be  known.  Based  upon  the  research 
hypothesis  of  this  study,  there  should  be  no  difference 
between  the  expected  final  costa  at  Tn  derived  from  the 
Martin  Cost  Model  using  the  degree  of  informational 
efficacy  present  at  TQ  and  the  actual,  final  costs. 
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While  at  Tft,  the  program  manager  also  plotted  line 


Cj,  over  time,  based  upon  the  estimated  cost  or  target 
price  of  385.  The  same  assumptions  as  mentioned  for  plot- 


ting line  C£,  were  applied.  There  is  a probability  of  .55 


that  the  target  price  of  385  will  be  the  actual,  final  cost 


at  T . Since  C„  at  385  is  the  target  price  at  which  the 
n I 


contract  was  awarded,  Cj  is  the  cost  line  that  the  program 
manager  will  monitor.  To  assist  the  program  manager  in 


monitoring  costs,  suppose  dash  lines  a and  b are  estab- 


lished as  tolerance  levels  on  either  side  of  the  cost  line. 


Cg.  As  long  as  the  actual  costs  occurring  over  time  stay 
within  the  interval  created  by  the  dash  lines  a and  b,  the 


costs  of  the  program  are  not  out  of  control.  However, 


in  this  case,  cost  line  C£  immediately  deviates  beyond  the 


interval  a,b  after  point  TQ;  an  out  of  control  situation 


potentially  exists  and  cost  growth  is  likely.  The  program 


manager,  therefore,  should  take  actions  which  include  an 


analysis  of  the  program  in  light  of  the  entropy  computa- 


tions made  at  T to  determine  areas  where  more  information 
o 


is  required,  thus  reducing  uncertainty  and  regaining 


control  over  costs. 


So  far,  the  discussion  of  the  use  of  the  Martin  Cost 


Model  has  been  limited  to  the  generation  of  an  expected 


final  cost  and  a cost  estimate  or  target  price  based  solely 


upon  informational  efficacy  provided  at  one  point  in 
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time.  That  time  is  at  the  initiation  of  the  development 
program.  The  economic  environment,  however,  contains 
uncertainties.  In  time,  these  uncertainties  could  present 
unexpected  events  that  affect  either  or  both  of  the  cost 
lines,  CE  and  Cj  (16:78).  At  the  time  that  the  program 
manager  becomes  aware  of  such  events  and  is  presented  with 
meaningful  and  useful  information  with  which  to  deal  with 
those  events,  the  cost  lines,  as  applicable,  could  be 
replotted  using  the  same  techniques  as  have  already  been 
described.  For  example,  consider  an  unexpected  technolog- 
ical breakthrough  at  time  interval  T^  which  has  caused  a 
major  revision  to  the  weapon  system  under  development.  In 
this  instance,  as  reflected  in  Figure  10,  both  cost  lines 
Cg  and  Cj  were  recalculated  and  replotted  along  with 
interval  a,  b,  using  the  level  of  informational  efficacy 
provided  at  . Also,  the  original  Cj  and  C£  lines,  based 
upon  the  informational  efficacy  provided  at  T0,  are  still 
shown  due  to  the  original  development  contract  being  under 
renegotiations . 

APPLICATION  OF  THE  MODEL 

Review  and  Significance  of  the  Model 

At  the  end  of  the  previous  section  to  this  Chapter, 
somewhat  simplistic  applications  of  the  Martin  Cost  Model 
were  presented  in  Figures  9 and  10.  These  examples,  how- 
ever, seem  to  imply  an  apparent  flexibility  and  adaptive- 
ness of  the  model  to  the  passage  of  time  and  to  a changing 
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environment.  The  underlying  continuum  for  the  model  i3 
time,  from  the  present,  which  incorporates  the  past,  and 
on  into  the  future  as  far  as  a given  estimate  is  required 
to  be  projected.  Further,  the  model  appears  to  be  able  to 
move  with  time.  As  the  future  becomes  the  present,  the 
model  can  again  be  applied  from  this  new  point  in  time. 

Within  the  time  continuum,  the  final  expected 
costs  of  the  program  and  the  alternative  cost  estimates 
and  target  price  can  be  evaluated  based  upon  a second  con- 
tinuum. The  second  continuum  involves  the  measurement  of 
the  certainty  and  uncertainty  or  risk  associated  with 
making  decisions  affecting  the  future  based  upon  the 
information  presently  provided.  To  the  extent  that  the 
information  is  meaningful  and  useful  to  the  decision- 
maker in  differentiating  alternatives  by  probability  of 

occurrence,  then  the  certainty  with  which  a decision  can 

\ 

be  made  is  increased  and  the  risk  of  error  reduced.  Thus, 
the  emphasis  of  the  model  is  upon  the  measurement  of  the 
present  level  of  informational  efficacy  within  the  decision- 
maker’s information  subsystem  and  the  relationship  of 
that  informational  efficacy  to  decision-making  in  terms  of 
the  management  and  control  of  the  future  with  respect  to 
potential  program  cost  growth  (16:3-40;  23:125-129) . 


Assumptions  of  the  Model 

Having  presented  the  conceptual  basis  for  the  model 
and  the  mathematical  techniques  required,  there  are  seven 


major  assumptions  which  can  be  derived. 


1.  The  conceptual  foundation  of  the  model  and  the 
predictions  generated  by  the  model  provide  a "descrip- 
tive" rather  than  "normative"  basis  upon  which  to 
analyze  cost  growth  at  the  macro  level  of  a development 
program. 


2.  The  expected,  final  cost  of  a development 
program  can  be  formulated  based  upon  the  ratio  of  cost 
estimate  or  target  price  to  the  level  of  informational 
efficacy  provided  in  a closed-looped  information 
subsystem  at  the  time  that  the  estimate  or  price  was 
established. 


3.  The  economic  environment  in  which  weapon  system 
acquisition  programs  function  may  reflect  perfect 
economic  competition  but  such  competition  is  deemed 
highly  improbable. 


4.  The  information  subsystem  of  the  program 
manager  can  be  characterized  as  a closed-loop  system 
which  possesses  both  a time  and  a certainty-risk- 
uncertainty  continuum.  The  certainty-risk-uncertainty 
continuum,  in  turn,  assumes  that: 


a.  Uncertainty  can  be  equated  to  risk  and  can 
be  measured  by  the  amount  of  entropy  contained  in  the 
information  subsystem. 


b.  Negentropy  can  be  equated  to  informational 
efficacy  which,  in  turn,  can  be  measured  as  the  remain- 
der of  system  unity  or  one  after  relative  entropy  has 
been  subtracted  from  unity.  Informational  efficacy 
further  equates  to  certainty  within  the  information 
subsystem. 


5.  Informational  efficacy  varies  inversely  with  the 
amount  of  entropy  contained  in  the  information  sub- 
system. 


6.  All  terras  reflecting  cost  or  price  in  the  model 
refer  to  an  economic  cost  consisting  of  both  the  DoD 
contractor's  costs  and  profits. 


7.  There  are  no  fund  limitations  imposed  directly 
upon  the  Martin  Cost  Model.  If  funding  limitations 
exist,  the  limitations  were  considered  by  the  informa- 
tion subsystem  at  the  time  that  cost  estimates  and 
target  price  were  formulated  as  alternatives. 


These  assumptions  are  basically  the  same  as  the  assump- 
tions presented  by  both  Martin  (23:125-127)  and  Glover  and 
Lenz  (16:33-34). 

Model  Limitations 

Besides  the  assumptions  listed*  tne  authors  of  this 
study  believe  that  there  may  presently  be  three  basic  limi- 
tations >.o  the  application  of  the  model.  First  of  all,  the 
model  was  specifically  developed  for  application  in  the 
development  phase  of  the  weapon  acquisition  program  (23:5) . 
The  only  known  validation  attempt  for  the  model,  as  far  as 
the  authors  have  been  able  to  research,  is  the  study 
accomplished  by  Glover  and  Lenz  (16) . The  Glover  and  Lenz 
study  also  applied  to  the  development  phase.  Therefore, 
until  such  time  as  further  validations  are  performed  to 
include  areas  other  than  the  development  phase,  the  appli- 
cation of  the  model  may  be  restricted  to  the  development 
phase  (16:35-36). 

A second  limitation  appears  to  be  in  the  alterna- 
tives provided  by  the  information  subsystem  (16:36).  If 
all  possible  alternatives  are  not  identified  by  the  infor- 
mation su’  item,  the  measurement  of  entropy  and  informa- 
tional efficacy  may  be  in  error.  These  errors  may  cause 
the  accuracy  of  expected,  final  costs  predicted  by  the 
model  to  decrease. 

The  third  limitation  is  that  the  Martin  Cost  Model 
was  formulated  a p.  part  of  a total  program  cost  model. 


Within  the  total  cost  model  (23:125-147) , the  costs  of 
providing  various  levels  of  informational  efficacy  are 
considered  in  conjunction  with  the  basic  Martin  Cost  Model 
as  presented  in  this  study  and  by  Glover  a'. id  Lenz  (16) . 

The  Martin  Cost  Model,  itself,  could  be  modified  to  include 
such  economic  variables  as  inflation.  Thus,  the  results 
obtained  from  this  study  and  that  of  Glover  and  Lenz  may 
not  be  indicative  of  the  actual  capabilities  of  the  Martin 
Cost  Model  if  applied  as  part  of  the  total  cost  model 
formulated  by  Martin. 

CONCLUSION 

If  the  basic  Martin  Cost  Model  can  be  sufficiently 
validated  for  application  within  the  development  phase, 
then  the  basic  model  may  form  the  foundation  for  predictive 
and  managerial  techniques  with  which  to  curtail,  at  least, 
that  portion  of  the  cost  growth  phenomenon  that  is  amenable 
to  control  by  the  program  manager. 


CHAPTER  III 
METHODOLOGY 

The  research  methods  applied  to  this  study  may  be 
divided  into  the  following  general  topics: 

1.  Literature  Review 

2.  Data  Collection 

3.  Data  Analysis 

4 . Data  Summary 

This  chapter  will  describe  the  procedures  that 
wore  used  and  the  results  of  data  collection  and  analysis 
of  the  Martin  Cost  Model.  Chapter  IV  presents  an  analysis 
summary  and  interpretation  of  the  findings  based  upon  the 
results  of  the  data  collection,  data  analysis,  and  test 
procedures.  Chapter  V presents  concluding  remarks  and 
recommendations  for  future  studies. 

LITERATURE  REVIEW 

The  purpose  of  the  literature  review  was  to  collect 
information  in  the  following  areas: 

1.  DoD  weapon  system  acquisition  process  and 
phases. 

2.  DoD  weapon  system  costs. 

3.  Cost  growth  and  coat  control  techniques. 
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4.  Decision  making  processes  and  techniques. 

5.  Communications  and  information  concepts, 
theories,  and  techniques. 

6.  Statistical  methods  and  application  as  they 
are  related  to  decision  making. 

7.  Research  data  techniques  and  methodologies. 
These  seven  areas  were  considered  the  major  areas  of 
concern  or  relevancy  in  developing  the  present  study. 
Although  the  Martin  Cost  Model  was  used  in  this  study, 
other  predictive  models  were  examined  during  the  review. 
The  results  of  the  literature  review  were  presented  in 
Chapters  I and  II. 

While  the  research  methodology  used  in  this  study 
can  be  divided  into  four  general  topical  areas,  the  basis 
fpr  the  detailed  expansion  of  the  methodology  employed 
rested  upon  the  results  of  the  literature  review  and  the 
authors*  decision  to  replicate  the  study  by  Glover  and 
Lem  with  subsequent  use  of  that  study  as  a guideline  for 
performing  the  research  effort  presented  here.  The 
remaining  sections  of  this  chapter  present  the  details  of 
the  data  collection  and  then  the  statistical  aud  testing 
procedures  used,  accompanied  by  the  results  obtained  by 
use  of  those  procedures. 


DATA  COLLECTION 


Overview 


The  collection  of  data  was  oriented  towards  the 
replication  of  the  Glover  and  Lenz  study  of  the  Martin  Coot 
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Model.  The  present  study  attempted  to  further  examine  the 
capability  of  that  model  to  accurately  predict  the  final 
development  costs  of  a weapon  system.  This  capability  was 
tested  using  the  data  obtained  from  the  F-5^(  TIGER  II, 
aircraft  weapon  system  development  program.  The  data 
obtained  was  separated  into  two  categories.  The  first 
category  included  the  data  to  be  used  in  determining  the 
entropy  or  uncertainty  that  existed  at  the  time  of  contract 
award  for  the  aircraft's  development.  The  second  category 
was  historical  in  nature  and  dealt  with  the  actual  costs 
incurred  during  the  development  phase.  This  cost  data  was 
used  in  performing  tests  of  the  cost  estimate  produced  by 
the  model  based  upon  the  data  provided  under  the  first 
category. 

Variables  Within  the  Study 

Both  data  categories  involve  consideration  of  four 
variables.  These  variables  correspond  to  the  four  concepts 
(uncertainty,  informational  efficacy,  costs,  and  time) 
that  are  integrated  by  Martin's  mathematical  model  into  a 
cost  prediction  technique.  In  the  following  paragraphs, 
time  and  cost  will  each  be  discussed  separately  while 
informational  efficacy  and  uncertainty  will  be  discussed 
from  the  standpoint  of  entropy. 


Time.  As  presented  in  Chapter  II,  time  was  described  as 
the  underlying  continuum  for  the  Martin  Cost  Model.  Time 
consists  of  the  present,  which  incorporates  the  past,  and 
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the  future  for  as  far  as  a given  cost  estimate  is  required 
to  be  projected.  The  passage  of  time  can  be  depicted  as 
presented  on  pages  58-60  of  Chapter  II.  Time  passes  in 
intervals  of  T.  Each  interval  was  one  calendar  quarter  or 
three  months  in  duration.  Time  started  at  TQ  which  repre- 
sents the  last  day  of  the  calendar  quarter  in  which  the 
development  contract  was  awarded  (31  December  1970)  and 
marks  the  beginning  of  the  development  program.  The  com- 
pletion of  the  program  is  designated  by  T^  or  the  last  day 
of  the  n^  quarter  (30  June  1974).  Calendar  quarters 
between  TQ  and  Tn  are  represented  by  T^ , reflecting  the 
last  day  of  the  ifc^  quarter. 

Cost.  In  Chapter  II,  the  term  cost  was  discussed.  In  this 
study,  cost  refers  to  the  economic  cost  of  the  weapon 
system's  development  to  include  the  contractor's  profit. 

As  reflected  in  Table  12,  Appendix  C (p.  126),  the  data  on 
the  cost  of  development  was  obtained  and  used  in  analysis 
by  calendar  quarters  or  time  intervals  of  T.  However,  the 
cost  data  presented  is  cumulative  by  quarter,  with  Tn  or 
the  14  time  interval  reflecting  the  final  cost  of  the 
development  program.  Costs  have  been  rounded  to  the 
nearest  thousand  dollars  and  are  reported  in  millions  of 
dollars.  For  example,  a figure  of  $10,112,020  would  be 
reflected  as  $10,112  million. 

The  specific  notation  to  be  used  in  conjunction 


with  the  cost  variable,  as  shown  in  Chapter  II,  is  restated 
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below: 

Cj  = Initial  cost  estimate  or  target  price  at 
the  tine  that  the  contract  was  awarded. 

Cg  = Expected  final  cost  of  the  program  as  predic- 
ted by  the  model . 

Added  to  this  notation  are  the  following  symbols: 

CA  = Actual  final  cost  ($89,322  million). 

Ci  = Quarterly  cumulative  cost  actually  incurred 
for  time  interval  t^. 


Entropy.  The  concept  of  entropy  was  discussed  earlier  in 
Chapter  II.  The  term  entropy  was  equated  with  the  uncer- 
tainty confronting  the  decision-maker  in  the  decision- 
making process.  Entropy  reflects  the  degree  of  disorder  or 
the  lack  of  organization  in  the  amount  of  information  pre- 
sented to  the  decision-maker  by  his  information  system  and 
from  which  he  is  to  select  an  alternative.  In  a given 
situation,  this  uncertainty  or  entropy  reaches  a maximum 
when  the  probability  of  each  alternative's  outcome  actually 
occurring  in  the  future  is  considered  to  be  equally  likely 
by  the  decision-maker.  To  calculate  the  relative  entropy 
(h)  of  a given  situation,  the  following  expression  is  used: 

n 

- pi  log  Pi 

h * ^'log'n and  lo9  n * “max 
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reflects  the  maximum  entropy.  The  n outcomes  are  assumed 
to  be  collectively  exhaustive  and  mutually  exclusive. 
Further,  the  probabilities,  P^,  are  constrained  as  follows: 
1.  0.0  * Pi  * 1.0 


n 


2. 


1.0 


i-1 

Informational  efficacy  (IE)  is  the  opposite  of  entropy  and 
reflects  both  the  order  within  the  situation  or  information 
system  and  the  degree  to  which  the  information  provided  is 
meaningful  or  useful  to  the  decision-maker . As  IE 
increases,  certainty  increases  and  the  amount  of  relative 
entropy,  h,  decreases.  The  degree  of  order  and  the  use- 
fulness of  the  information  provided  can  be  expressed  by: 


IE  » 1 - h 


Both  of  these  calculations,  h and  IE,  are  critical 
to  the  generation  of  cost  predictions  by  the  model.  Obvi- 
ously, to  produce  meaningful  calculations,  a data  base  is 
required.  That  base  shall  be  discussed  next. 


Source  of  Data 

Since  a study  involving  a development  program  which 
was  just  getting  underway  would  have  been  both  beyond  the 
scope  and  time  constraints  of  the  present  study,  the  data 
base  for  this  study  rested  upon  the  selection  of  a recently 
completed  development  program  as  discussed  in  Chapter  I. 

The  program  selected  was  the  F-5E,  TIGER  II,  Aircraft 
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Development  Program.  The  System  Program  Office  (SPO)  for 
the  P-5E  is  located  at  Wright-Patterson  Air  Force  Base, 
Ohio. 


Aside  from  considerations  of  scope  and  time,  the 
authors'  selection  of  the  F-5E  as  the  data  source  was  based 
upon  the  following  reasoning: 

1.  Although  the  program  did  not  deal  with  the 
development  of  a new  weapon  system,  per  se,  the  pro- 
gram did  include  major  modifications  to  an  aircraft 
which  reflected  technological  advancements. 

2.  The  desired  aircraft  modifications  were  assumed 
to  be  sufficiently  within  the  state  of  the  art  at  the 
time  that  the  development  contract  was  awarded  so  as 

to  preclude  the  generation  of  almost  complete  uncer- 
tainty while  yet  avoiding  almost  complete  certainty  on 
all  aspects  of  the  development.  Thus,  it  was  hoped 
that  the  development  of  the  aircraft  would  not  involve 
an  examination  of  the  model  at  the  extremes  of  the 
certainty-risk-uncertainty  continuum,  but,  instead, 
would  be  more  aligned  with  the  situation  found  by 
Glover  and  Lenz  in  which  relative  entropy  was  computed 
to  be  0.686  (16:53) . 

3.  The  magnitude  of  the  dollar  value  involved  in 
the  contract,  $83,635,000  was  considered  substantial 
by  the  authors  and  thus  would  not  represent  a trivial 
application  of  the  Martin  Cost  Model. 

4.  Both  the  SPO  and  data  locations  were  in  the 
proximity  of  the  research  center  (16:46).  This 
physical  closeness  was  expected  to  ease  the  research 
effort  while  reducing  any  costs  involved  in  that  effort. 

5.  Personnel  who  had  been  in  key  managerial 
positions  during  the  development  phase  were  still 
located  at  the  SPO. 

6.  The  development  phase  had  only  recently  been 
completed  (30  June  1974) . Data  were  readily  available 
and  the  experiences  of  having  worked  on  the  program 
should  still  be  somewhat  fresh  in  the  minds  of  the 
key  personnel  within  the  SPO. 


75 


Specific  Sources  for  Entropy  (Uncertainty)  and  Informa- 
tional Efficacy.  Having  selected  the  F-5E,  TIGER  II,  for 
the  data  base,  the  authors  contacted  the  United  States  Air 
Force  Business  Research  Management  Office  located  at  Wright- 
Patterson  Air  Force  Base,  Ohio.  Through  the  assistance 
rendered  by  the  Center,  the  authors  readily  gained  access 
to  the  F-5E  SPO  Director.  The  authors  obtained  approval 
from  the  Director  to  use  information  about  the  F-5E  develop- 
ment phase  for  the  purposes  of  this  study.  At  the  authors' 
request,  the  Director  also  furnished  the  names  of  six 
personnel  (participants)  within  the  SPO  whom  he  believed 
had  been  in  various  managerial  positions  directly  involved 
during  the  development  of  the  F-5F..  As  shall  be  further 
discussed  in  subsequent  paragraphs,  the  authors'  selection 
of  a directed  interview  technique  using  the  DELPHI  procedure 
as  a means  of  data  collection  necessitated  the  indentifica- 
tion  and  involvement  of  the  participants  in  this  study. 

Prior  to  the  actual  collection  of  data,  the  authors 
contacted  each  prospective  participant  to  elicit  their 
cooperation  in  the  study,  to  establish  times  for  the  inter- 
views, and  to  determine  whether  or  not  the  prospective 
participant  had  in  fact  been  in  a managerial  position 
directly  involved  during  the  development  phase.  Three  of 
the  participants  were  found  to  have  been  involved  while 
three  were  not  involved  during  the  development  phase.  The 
three  that  had  not  been  directly  involved  were  deleted 
from  further  consideration  within  the  study.  One  of  the 
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three  prospective  participants  that  was  removed  from  the 
study,  however,  did  provide  the  authors  with  the  name  of  a 
seventh  prospective  participant.  This  seventh  individual 
was  contacted  and  was  found  to  have  been  involved  as  a 
manager.  The  data  base  was  initially  to  be  gathered  from 
four  managerial  participants.  Since  anonymity  had  been 
promised  by  the  authors  during  the  elicitation  of  their 
cooperation,  there  can  be  no  further  description  of  the 
participants  provided  except  as  follows: 

1.  Each  had  been  in  a managerial  position  during 
the  development  phase.  One  participant  was  positioned 
within  the  Engineering  section  of  the  SPO,  another 
within  the  Management  section,  and  two  within  the 
Financial  Management  section. 

2.  Three  of  the  participants  had  Government  Serv- 
ice ratings  of  13  at  the  time  while  the  fourth  had 
been  a second  lieutenant  within  the  United  States  Air 
Force. 


3.  Three  of  the  participants  had  been  in  cne  or 
more  managerial  positions  from  the  on-set  of  the 
development  phase.  The  fourth  had  managerial  exper- 
ience directly  related  to  the  program  only  after  the 
development  phase  had  begun.  However,  this  fourth 
participant  indicated  to  the  authors  and  to  the  authors' 
satisfaction  that  he  was  well  familiar  with  the 
development  phase  and  could  provide  an  adequate  source 
of  information. 

Although  the  data  base  was  initially  formulated 
around  four  participants,  one  participant  in  the  SPO 
Management  section  was  subsequently  deleted  from  the  study. 
His  removal  was  due  to  an  unexpected  temporary  duty  assign- 
ment to  another  area  of  the  country.  Since  the  assignment 
was  projected  to  last  from  7 to  14  days,  the  authors  felt 
that  under  the  DELPHI  procedure  being  used  the  time  delay 
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could  have  adversely  affected  the  collection  of  data  from 
all  participants  (4:62).  Therefore,  the  participant  was 
removed,  but  the  data  that  had  been  collected  up  until  the 
time  of  his  departure  was  retained  within  the  study  and 
used  in  conjunction  with  further  data  collection  from  the 
remaining  three  participants.  For  further  explanation 
pertaining  to  this  removal,  refer  to  Appendix  A (p.  105). 

While  the  current  study  is  basically  a replication 
of  the  Glover  and  Lenz  study,  the  authors  assumed  that  the 
data  base  formed  by  the  participants  would  be  sufficient 
for  the  purposes  of  this  study.  Glover  and  Lenz  had  not 
only  collected  data  from  participants  using  the  DELPHI 
procedure  but  had  also  reviewed  weapon  system  program  docu- 
ments including  source  selection  data  (16:47).  However, 
such  a review  was  assumed  to  be  unnecessary  by  the  authors, 
since  a detailed  analysis  of  a specific  development  pro- 
gram is  not  the  intent  of  the  research  effort  and  neither 
the  Martin  Cost  Model  nor  the  DELPHI  procedure  appear  to 
require  such  a review. 


Interview  and  DELPHI  Procedure.  By  virtue  of  its  defini- 
tion, the  data  necessary  to  evaluate  entropy  was  not 
expected  to  be  found  in  a single  report,  document,  or 
individual.  The  objective  was  to  collect  applicable  data 
that  was  in  exit. mce  at  the  time  of  cc-Tact  award. 

To  calculate  entropy  as  formulated  in  Chapter  II, 
it  was  necessary  to  collect  data  regarding  possible  program 


outcomes  and  the  probability  of  each  outcome.  To  accom- 
plish this  collection  task,  a process  used  by  the  RAND 
Corporation  for  the  elicitation  of  expert  opinion,  called 
the  DELPHI  procedure,  appeared  to  be  suitable  for  the  pur- 


pose. 


The  DELPHI  procedure  is  a method  to  elicit  the 


opinions  of  experts  (6)  using  a carefully  designed  question- 
naire administered  in  a series  of  iterative  rounds  of 
questioning  (9)  in  the  form  of  a directed  interview. 
Associated  with  each  round  of  questioning  is  a feedback  of 
information  gathered  from  the  previous  round  (10) . The 
feedback  and  the  subsequent  data  gathered  is  refined  in 
each  successive  iteration  (7) . 

The  DELPHI  procedure  has  certain  desirable  advan- 
tages (19:20-23): 

1.  Concensus  reflects  reasoned,  self-aware  sub- 
jective valuations  expressed  in  the  light  of  subjective 
evaluations  of  associate  experts. 

2.  Controlled  feedback  makes  group  estimates  more 
accurate . 

3.  Procedures  create  a well-defined  process  that 
can  be  described  quantitatively. 

The  DELPHI  procedure  is  an  alternative  to  the 
committee  approach  for  eliciting  a group  judgment.  It 
improves  upon  the  committee  approach  by  subjecting  individ- 
ual expert  views  to  criticism  without  face-to-face  con- 
frontation. It  provides  anonymity  of  opinion  (4:44). 

Thus,  it  attempts  to  improve  upon  the  committee  approach 
oy  allowing  the  exchange  of  information  in  an  environment 
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of  reduced  group  pressure  to  conform  and  remove  the  impact 
of  the  dominant  individual  (4:63). 

For  this  study,  a series  of  questions,  in  three 
rounds,  was  developed,  via  the  DELPHI  procedure  to  gather 
the  entropy  data.  The  first  round  responses  formed  the 
basis  for  each  successive  round  by  identifying  features 
considered  to  be  uncertain  as  to  outcome.  The  specific 
set  of  questions  is  detailed  in  Appendix  A together  with 
specific  response  summaries.  The  questions  were  reviewed 
and  approved  by  members  of  the  Air  Force  Institute  of 
Technology  faculty  for  sufficiency  with  respect  to  expected 


results. 

The  aspects  shown  in  Appendix  A,  Summary  of  Round  I 
Responses  (p.  103},  represent  all  the  uncertain  areas  of 
the  I-5E  program  as  identified  by  collective  opinion  of 
the  group  of  experts  participating  under  the  DELPHI  pro- 
cedure. In  Appendix  A,  Tables  6 and  7 (pp.  106-110) , the 
number  in  each  cell  represents  the  arithmetic  average  of 
the  responses  for  that  particular  probability  in  the  cell. 
These  data  were  used  to  compute  the  value  of  entropy  out- 
lined in  a subsequent  section  of  the  chapter. 


DATA  ANALYSIS 


Cost  Prediction  Data 


The  data  collected  for  the  measurement  of  entropy, 
via  the  DELPHI  procedure,  were  reduced  to  two  key  variables; 
n,  or  the  number  of  possible  outcomes  for  the  F-5E 
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development  program;  and  p^,  or  the  probability  associated 
with  the  ifck  outcome.  Using  the  DELPHI  procedure,  there 
were  12  aspects  identified  as  uncertain  in  outcome  and 
three  feature  outcomes  for  each  aspect.  The  probabilities 
associated  with  the  outcome  aspects  are  listed  later  in 
Table  7,  Appendix  A (pp.  109-110) . 

Since  none  of  the  participants  responded  to  the 
feature  outcome  category  labeled  "Unacceptably  Excellent," 
this  category  was  removed  from  further  consideration  in 
determining  possible  program  variances.  Therefore,  a 
program  outcome  was  defired  as  a set  of  12  aspects,  each 
associated  with  one  of  the  2 possible  feature  outcomes  in 
this  particular  research  and  shown  in  Table  8 , Appendix  A 
(p.  111).  This  definition  produced  a total  4,096  possi- 
ble program  outcomes.  The  probability  of  a program  outcome 
was  defined  as  the  product  of  the  probabilities  of  one 
feature  outcome  taken  from  each  of  the  12  aspects  associ- 
ated with  the  program  outcome,  assuming  independence  among 
the  aspects  (refer  to  Appendix  B,  p.  113,  for  detailed 
discussion  on  the  derivation  of  these  outcomes) . 

To  compute  the  numerical  value  of  entropy,  a short 
FORTRAN  IV  computer  program  developed  by  Glover  and  Lenz, 
was  used.  The  program  is  discussed  and  listed  in  Appendix 
B (pp.  115  and  116  respectively) . The  results  of  the 
calculation  for  the  value  of  entropy  are: 
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Martin  Cost  Model  Prediction 


To  use  the  Martin  Cost  Model  to  predict  final  total 
cost  of  the  F-5E  development  program,  the  formula  is: 

CT 


CE  = 1 - h 


(23:126) 


A cost  forecast  was  made  at  T0  using  information  developed 
during  the  analysis.  Cj  is  the  initial  contract  target 
price  taken  from  the  development  contract.  Relative 
entropy,  h,  is  calculated  with  information  derived  from  the 
DELPHI  procedure  concerning  expected  outcomes  at  Tn,  pro- 


jected from  Tq.  The  calculation  of  C£  is  as  follows: 


83.635 


83.635 


E 1 - h 1 - .91  .09 

where  1 - h = IE, 

CE  * $929,278  million. 

Thus,  the  estimated  total  program  cost  is  $929,278  million 
as  compared  to  an  actual  program  development  cost  of 
$89,322  million.  This  estimated  total  program  development 
cost,  CE,  represents  a figure  10.4  times  greater  than  the 
actual  total  program  development  cost. 

It  is  evident  from  a comparison  of  the  actual  to 
estimated  total  program  development  cost,  a great  differ- 
ence, that  the  Martin  Cost  Model  did  not  work  as  designed. 
With  a decrease  in  informational  efficacy,  program  costs 
increace.  Based  on  the  outcome  of  the  DELPHI  procedure 
and  the  probabilities  assigned  by  the  participants,  a 
relative  entropy  of  .91  was  calculated  in  the  F-5E  study. 
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Since  relative  entropy,  h,  is  a measure  of  the  degree  of 
uncertainty  in  a situation,  this  outcome  indicates  that  an 
extremely  high  degree  of  uncertainty  does  exist. 


DATA  SUMMARY 


This  chapter  presented  the  methodology  and  specific 
results  associated  with  data  collection  and  test  performed 
with  respect  to  the  research  hypothesis.  Actual  cost  data 
from  the  F-5E,  TIGER  II,  development  program  was  compared 
to  the  cost  prediction  generated  by  the  Martin  Cost  Model. 
Based  on  the  compared  data,  the  Martin  Cost  Model  does  not 
even  approach  the  figure  for  actual  program  development 
cost  but  exceeds  it  by  over  900%.  A further  explanation 
of  these  results  will  be  discussed  in  Chapter  IV.  From 
the  results  obtained  thus  far,  the  research  hypothesis  is 
not  supported. 


CHAPTER  IV 


ANALYSIS  AND  SUMMARY  OF 
FINDINGS 

Chapter  III  presented  the  procedures  or  methodology 
used  for  data  collection/  data  analysis/  and  the  test  of 
the  Martin  Cost  Model.  The  presentation  also  included, 
based  upon  that  methodology,  the  calculation  of  the  values 
required  for  testing  and  analyzing  the  ability  of  the  model 
to  accurately  forecast  or  predict  the  final  development 
cost  for  the  F-5E,  TIGER  II,  aircraft.  In  Chapter  III,  the 
findings  were  the  opposite  of  what  was  expected,  as 
reflected  in  the  research  hypothesis.  Several  areas  were 
explored  to  explain  the  poor  predictive  power  of  the  model. 

As  presented  in  Chapter  III  and  Appendix  C,  the 
research  hypothesis  of  the  current  study  was  not  supported. 
The  mathematical  calculations  of  Appendix  B along  with  the 
study  comparisons  of  Appendix  D reveal  the  Martin  Cost 
Model  does  not  hold  under  the  current  situation.  Many 
factors  contribute  to  the  failure  of  the  research  hypoth- 
esis, but  most  significant  was  the  difference  of  opinion 
and  resultant  assignment  of  probabilities  to  the  alterna- 
tives summarized  in  DELPHI  Round  I.  The  results  of  this 
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probability  assignment  produced  a high  entropy  calculation 
yielding  a low  informational  efficacy.  Appendix  E pro- 
vides a comparison  of  participant  answers  and  an  analysis 
of  the  resultant  dispersion,  A discussion  of  other  types 
of  cost  adjustment,  the  need  for  a prior  facility  with 
probability,  and  the  DELPHI  procedure  itself  i3  contained 
in  this  chapter. 


APPLICATION  OF  COST 
ADJUSTMENTS 

Within  the  Glover-Lenz  study,  consideration  was 
given  to  possible  adjustments  to  the  actual  final  cost  of 
the  development  program  prior  to  applying  the  least-squares 
method  of  fitting  a straight  line  through  the  cost  obser- 
vation data,  cT  (16:64).  Such  adjustments  would  have  been 
based  upon  changes  in  the  actual  costs  which  were  Attrib- 
utable to  " . . . causes  external  to  the  program  . . . 
[16:64],"  as  in  the  case  of  abnormal  cost  escalations  or 
externally  directed  changes  to  the  weapon  system's  capa- 
bility (16:64).  Glover  and  Lenz  concluded,  however,  that 
since  the  points  in  time  at  which  such  externally  caused 
changes  in  cost  should  be  removed  could  not  be  determined 
and  since  such  adjustments  would  have  had  "...  relatively 
small  influence  . . . [16:64],"  the  unadjusted  final  co3t 
figure  was  retained. 

While  the  same  conclusion  was  assumed  to  be  appli- 
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minor  importance.  Though  there  are  methods  available  to 
take  cost  adjustments  into  consideration , they  were  not 
utilized.  Instead;  adjustments  to  the  final  cost  of  the 
development  program  were  not  considered  appropriate  in  the 
present  study  due  to  the  large  difference  between  tne 
estimated  final  development  cost  and  the  actual  final 
development  cost/  a difference  of  over  900%. 

BACKGROUND  EXPERIENCE 
OP  EXPERTS 


In  the  weapon  system  acquisition  process,  the 
opinion  of  experts,  based  on  their  personal  experience,  is 
critical  to  the  measurement  of  risk  and  uncertainty.  These 
factors  of  risk  and  uncertainty,  as  measured  by  expert 
experience,  contribute  to  the  subjective  probability  that 
an  event  will  occur.  Expert  opinion  is  based  on  experience, 
information,  and  intuition.  This  information,  along  with 
the  manipulation  of  the  information  process  yielding  the 
calculation  of  entropy  are  two  key  concepts  of  the  Martin 
Cost  Model. 

In  the  manipulation  of  the  Martin  Cost  Model,  a 
certain  knowledge  of  and  facility  with  probability  and  the 
potential  impact  of  risk  and  uncertainty  on  the  final 
development  program  cost  are  necessary  for  accurate  pre- 
dictions, This  basic  background  is  required  to  give  the 
participants  an  understanding  of  the  technique  attempted. 
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The  participants  must  "feel  involved"  in  the  study  in 


order  to  provide  complete  cooperation  on  a voluntary  basis. 


CURRENT  DELPHI  PROCEDURE 
STATUS 


The  DELPHI  procedure  requires  that  participating 


panelists  be  experts  in  the  subject  area  and  that  a con- 


sensus be  obtained  through  reliable  and  valid  procedures 


(24:4).  However,  Henry  M.  Parsons,  in  a 1972  study,  con- 


cluded that  the  reliance  of  system  designers  or  the 


opinions  and  preferences  of  ".  . . experts"  is  "foolhardy." 


Such  experts  "...  may  provide  suggestive  leads , but  are 


not  reliable  guides  as  demonstrated  by  repeated  disagree- 


ment with  objective  data  . . . [24: 17).' 


Another  feature  that  must  be  considered  is  whether 


the  group  responses  can  be  aggregated  meaningfully,  If 


there  is  no  way  to  meaningfully  aggregate  group  response. 


then  the  DELPHI  procedure  would  probably  have  questionable 


results.  External  pressures  to  conform  to  "popular"  or  top 


level  management  decisions  or  a desire  to  avoid  "rocking- 


the-boat"  may  seriously  affect  the  DELPHI  procedure  (4) . 


Part  of  the  basic  foundation  of  the  DELPHI  is  the 


guarantee  of  anonymity  of  participants  involved.  This 


facet  serves  the  purpose  of  attracting  expert  panelists  by 


guaranteeing  protection  against  individual  accountability 


and  an  invitation  to  "permissive  brainstorming  where 
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anything  goes."  Lacking  accountability,  a participant 
can  blame  nameless  others  for  any  findings  hs  does  not 
like  (24:62). 

Inherent  in  the  DELPHI  are  both  independence  and 
dependence  between  experts  which  is  at  the  heart  of  the 
DELPHI  iteration  "with  feedback."  The  first  round  of 
questions  is  designed  to  secure  independent  expert  judgment. 
Successive  rounds  provide  correlated  or  biased  judgments 
(24:18).  Rationalizations  concerning  reconsiderations, 
incorporation  of  new  information,  and  convergence  toward 
consensus  cannot  hide  the  fact  that  independent  judgment 
is  destroyed  once  the  participants  know  how  others 
responded  (24:18).  The  DELPHI  deliberately  manipulates 
responses  towards  minimum  dispersion  of  opinion  in  the  name 
of  consensus  (24:47).  By  the  time  a third  or  fourth  round 
occurs,  the  holdout  individualist  sees  the  possibility  of 
another  round  of  questioning  and  yields  to  save  another 
round  (24:49). 

Complex  future  events  do  not  lend  themselves  to 
clear  and  unambiguous  descriptions  typical  of  the  one- 
sentence  DELPHI  questionnaire  format.  Vague,  generalized 
description#  of  future  events  permit  respondents  to  project 
any  one  of  a large  number  of  possible  scenarios  as  his 
interpretation  of  the  event.  Verbal  responses,  when  they 
occur,  are  vague  with  sweeping  descriptions,  slogans,  or 
simplistic  statements.  The  structure  and  dynamics  of  the 
DELPHI  response  contribute  to  compounding  ambiguity  (24:49). 


DELPHI  items  are  typically  broad,  amorphous  classes 
of  events  not  precisely  defined.  DELPHI  forecasts  are 
opinions  about  these  broad  classes  of  events,  not  syste- 
matic predictions.  Opinions  are  typically  snap  judgments 
based  on  free-association  stereotypes.  Opinion  consensus 
tends  to  be  manipulated  to  minimize  opinion  dispersion. 

The  DELPHI  procedure  produces  transient  attitudes  about  the 

future,  which  xs  quite  different  from  systematic  predictions 

of  the  future.  The  DELPHI  questionnaire  format  does  not 

lend  itself  to  scientifically  objective  and  externally  ' 

verifiable  statements  of  future  events  (24:61). 

Accuracy  can  not  be  measured  for  most  DELPHI  ■ 

items,  because  changing  attitudes  and  opinions  on  ' 

amorphous  issues  are  not  true  or  false  and  do  not 
have  specific  dates  at  which  they  occur  [24:61]. 

There  is  nothing  basically  wrong  with  studying  and  learning 

more  about  opinions  concerning  the  future.  We  should  not  j 

t 

confuse  such  opinions,  however,  with  seriously  considered, 

\ 

qualified  and  documented  predictions  of  well-defined  future 
developments.  Attitudes  and  opinions  change;  fresh 
sampling  in  real  time  is  needed  to  track  these  changes. 

Sampling  must  be  explicit  in  terms  of  subject  populations 
if  any  systematic  inferences  are  to  be  made  (24:61). 

The  originators  of  the  DELPHI  procedure  had  the 
right  idea  in  answer  to  a pressing  need  to  enlist  the  aid 
of  geographically  separated  experts  to  work  together  on 
unknown  and  complex  problems.  One  of  the  most  significant 
aspects  of  their  work  was  the  accumulation  of  knowledge 


through  a series  of  interactions  and  feedback.  This 
concept  answered  the  social  need,  but  implementation  has 
been  counterproductive.  Instead  of  testing  numerous 
alternatives,  the  method  zeroed  in  on  iterative  group 
response  (24:72). 


SUMMARY 


As  applied  to  the  current  study,  the  DELPHI 
procedure  did  not  provide  evidence  to  substantiate  the 
ability  of  the  Martin  Cost  Model  to  predict  final  program 
development  costs.  In  actuality,  the  opposite  effect  of 
what  had  been  the  research  hypothesis  occurred  and  the 
research  hypothesis  was  not  supported. 

In  analyzing  the  data  obtained  from  the  F-5E  par- 
ticipants, a significant  difference  of  r pinion  and  proba- 
bility assignment  to  cited  areas  of  uncertainty  were  noted. 
This  may  have  resulted  for  numerous  reasons  ranging  from 
a lack  of  facility  for  probability  assignment  through  a 
failure  to  believe  anonymity  would  be  maintained  coupled 
with  a Hdo  not  rock-the-boat"  attitude.  Based  on  these 
aspects  and  others  previously  mentioned  in  Chapter  IV,  the 
DELPHI  procedure  leaves  much  to  be  desired.  It  appears  to 
be  a poor  tool  to  obtain  data  for  use  with  the  Martin  Cost 
Modal,  Some  other  technique  or  combination  of  techniques 
should  be  investigated  to  determine  if  one  exists  which  is 
capable  of  providing  accurate  and  readily  useable  informa- 
tion for  use  with  the  Martin  Cost  Model. 
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CHAPTER  V 


4 


CONCLUSIONS  AND  RECOMMENDATIONS 


Within  this  chapter,  the  fi:  al  conclusions  for  the 
current  study  will  be  presented.  Following  the  conclusions 
will  be  the  recommendations  for  future  research  into  the 
area  of  testing  the  validity  of  the  Martin  Cost  Model  as  a 
predictor  of  final  development  coat  and  as  a tool  with 
which  to  predict  and  control  cost  growth. 


CONCLUSIONS 


V 

r* 


In  fulfilling  the  national  policies  and  objectives 
of  the  United  States,  the  DoD  must  insure  that  military 
weapon  system  sophistication  is  maintained  and  increased 
as  technology  advances  throughout  the  world.  The  resources 
of  society,  however,  are  limited.  Competition  for  the 
resources  that  are  available  have  increased  between  the 
various  sectors  of  society.  The  DoD  is  in  competition 
with  the  growing  public  demands  for  non-military  related 
governmental  programs  and  services.  Granted,  total  federal 
expenditures  have  increased.  However,  in  terras  of  available 
dollars,  the  proportion  that  has  been  allotted  to  the  DoD 
for  military  purposes  has  continued  to  shrink  while  the 
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costs  of  maintaining  and  upgrading  weapon  system  sophisti- 
cation has  grown  at  a disproportionate  rate.  To  continue 
to  fulfill  national  defense  policies  and  objectives,  ways 
must  be  found  to  judiciously  manage  those  limited  resources 
that  are  allocated  to  the  DoD.  One  avenue  available  to 
manage  allotted  resources  is  to  develop  a method  to  curtail 
cost  growth,  in  military  weapon  system  acquisition  programs. 

The  subject  of  the  current  study,  the  Martin  Cost 
Model,  is  a recent  attempt  to  formulate  a technique  to  deal 
with  potential  and  existing  cost  growth.  The  Martin  Cost 
Model  is  a mathematical  model  into  which  such  concepts  as 
entropy,  informational  efficacy,  cost,  and  time  have  been 
integrated  for  the  purpose  of  providing  a predictive  mana- 
gerial tool  for  weapon  system  development  cost.  The 
application  of  the  model  to  a development  program  rests 
upon  certain  assumptions  referent  to  cost  growth  and 
weapon  system  development  program  management.  These 
assumptions  are  listed  here  in  summary  form: 


1.  The  cost  growth  definition  assumes  no  change  in 
quantity  produced. 


2.  The  program  manager,  contractor,  and  associated 
information  form  an  information  system. 


3.  A condition  of  total  information  m a system 
equates  to  unity. 


4.  The  program  manager  is  experienced. 


5.  The  Martin  Theory  is  descriptive,  not 
normative . 


6.  The  effective  program  cost  may  be  represented 
by  a ratio  of  target  price  to  informational  efficacy  of 
the  data  in  a closed  decision  system. 
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7.  Entropy  is  a measure  of  disorder  in  an  informa- 
tion system. 

8.  The  economic  environment  of  a weapon  system 
acquisition  program  may  reflect  perfect  economic  compe- 
tition but  it  is  deemed  highly  improbable. 

9.  The  information  subsystem,  characterized  by 
a closed  loop  system,  assumes: 

a.  Uncertainty  can  be  equated  to  risk  and 
measured  by  the  amount  of  entropy  in  the  information 
subsystem. 

b.  Negentropy  can  be  measured  as  the  remainder 
when  the  value  of  relative  entropy  is  subtracted  from 
unity. 

10.  Informational  efficacy  varies  inversely  with 
information  in  the  information  subsystem. 

11.  All  terras  reflecting  cost  or  price  in  the 
model  refer  to  economic  cost  consisting  of  the  contrac- 
tor's costs  and  profits. 

12.  There  are  no  limitations  imposed  directly  on 
the  Martin  Cost  Model. 

If  these  assumptions  are  correct,  then  the  basis  for  pre- 
diction and  managerial  control  resta  in  the  measure  of 
entropy.  This  implies  the  ability  of  the  program  manager's 
information  system  to  identify  all  possible  alternative 
program  outcomes  and  accurately  assess  the  alternatives  in 
probabilistic  terms  does  exist.  To  the  extent  that  such 
abilities  are  inadequate,  the  usefulness  of  the  model  as  a 
tool  rapidly  declines. 

The  question  as  to  whether  the  Martin  Cost  Model  is 
a viable  technique  for  predicting  and  managing  final  weapon 
system  development  cost  or  not  still  remains  unanswered. 

On  the  surface,  the  results  of  the  present  study  indicate 
that  this  model  has  failed  to  predict  and  manage  cost.  The 
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model  predicted  a final  development  cost  for  the  F-5E, 

TIGER  II,  aircraft  which  was  10,4  times  greater  that  the 
actual  final  cost.  The  initial  contract  price  for  develop- 
ment was  within  6.37%  of  the  final  coat.  The  conclusions 
of  this  study  are  the  exact  opposite  of  that  reached  in  the 
Glover-Lenz  study  even  after  the  discrepancies  in  data  pre- 
sentation and  inappropriate ness  of  the  statistical  test 
employed  are  considered  as  shown  in  Appendix  C and  D. 

In  the  Glover-Lenz  study,  the  Martin  Cost  Model 
was  found  to  be  a viable  technique.  The  model  predicted  a 
final  development  cost  for  the  SRAM  which  was  only  3.9% 
greater  than  the  actual  final  cost.  Aside  from  the  con- 
clusions drawn  from  the  two  studies,  the  only  major  differ- 
ence is  one  of  situation.  While  both  studies  used  the  same 
basic  data  collection  technique  centered  around  the  DELPHI 
procedure,  the  data  base  for  one  study  rested  upon  the 
development  phase  of  the  SRAM  missile  while  the  other 
study  rested  upon  the  F-5E,  TIGER  II,  aircraft.  The 
answer  to  the  model  viability  question  appears  to  reside  in 
determining  the  utility  of  the  concept  of  entropy  upon 
which  the  cost  prediction  of  the  model  rests.  The  utility 
can  be  determined  only  through  further  research  into  dis- 
covering and  applying  a means  of  measuring,  with  accuracy, 
the  entropy  of  the  model  in  a given  development  program. 

The  disparity  between  the  results  of  the  two  studies  lead 
to  the  conclusion  that  the  DELPHI  procedure  is  not  univer- 
sally applicable  to  all  situations. 
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RECOMMENDATIONS 


On  a conceptual  level,  the  Martin  Cost  Model  appears 
to  be  a potentially  viable  technique  for  predicting  and 
controlling  weapon  system  development  costs.  The  diffi- 
culty, however,  comes  in  attempting  to  apply  the  model.  As 
indicated  previously,  the  difficulty  centers  upon  determin- 
ing the  utility  of  the  concept  of  entropy  as  a basis  for 
predicting  cost  and  measuring,  with  accuracy,  the  entropy 
involved  in  a development  situation.  If  specific  means 
for  making  such  a measurement  had  been  available,  the 
results  of  the  current  study  might  have  been  different.  As 
a comparison  of  the  results  of  the  two  studies  has  shown, 
the  model  can  yield  dramatically  different  results.  This 
does  not  mean  that  the  Martin  Cost  Model  should  be  ignored 
but,  rather  suggests  further  study  is  appropriate  with 
emphasis  on  the  following  areas: 

1.  Expand  the  research  effort  to  define  and 
develop  some  way  other  than  the  DELPHI  procedure  to 
measure  outcome  probability. 

2.  Based  on  the  results  of  the  above  research,  a 
decision  should  be  made  as  to  the  feasibility  of  apply- 
ing the  model  in  a real  world  context.  If  the  model 

is  to  be  applied  and  the  results  analyzed,  a historical 
frame  of  reference  undoubtedly  will  be  used  since 
application  of  the  model  to  an  on-going  development 
program  would  take  years  to  complete.  In  using  actual 
historical  data,  however,  greater  efforts  should  be 
taken  to  obtain  information  about  the  detailed 
circumstances  surrounding  the  actual  development  pro- 
gram, With  such  information,  it  would  be  possible  to 
perform  a more  valid  test  of  the  predictive  accuracy 
of  the  model. 


3.  In  studying  and  applying  the  model,  further 
research  should  be  devoted  to  developing  statistical 
procedures  for  testing  the  model. 


SUMMARY 


The  present  study  has  cast  doubt  on  the  usefulness 
of  the  Martin  Cost  Model  as  a predictive  and  managerial  tool 
with  which  to  address  cost  associated  with  the  development 
stage  of  a weapon  system.  The  final  determination  as  to 
the  usefulness  of  the  model  rests  in  further  analysis  of 
the  utility  of  the  concept  of  entropy  or  uncertainty  as  it 
relates  to  decision  making  and  the  occurrence  of  cost  plus 
the  development  of  a means  to  accurately  measure  entropy. 
Regardless  of  the  ultimate  disposition  of  the  Martin  Cost 
Model,  it  should  be  recognized  that  the  model  has  already 
been  of  some  value.  It  has  reemphasized  that  the  determi- 
nation and  management  of  development  program  costs  are 
functions  of  the  quality  or  adequacy  of  the  information 
system  upon  which  the  program  manager  bases  his  decisions. 


APPENDIX  A 


COLLECTION  OF  DATA  FOR 
MEASURING  ENTROPY 


This  appendix  presents  the  questions  and  responses 
of  the  DELPHI  procedure  explained  in  Chapter  III.  The 
DELPHI  procedure  would  normally  involve  four  rounds  of 
interrogation  (6:6)  but,  due  to  responses  received  in 
Round  III,  Round  IV  was  not  considered  necessary.  The 
following  sections  present  the  questions  and  responses  for 
each  individual  round. 


ROUND  I 


In  Round  I,  the  scenario  was  established  which 
was  used  for  the  remaining  rounds  of  interrogation. 


Round  I Questions 

The  following  questions  were  used  in  Round  is 

Question  1.  List  those  aspects  - technical  charac 
teristics,  performance  parameters,  physical  character- 
istics - for  the  F-5E  development  phase,  about  which 
you  felt  the  outcome  could  have  been  something  other 
than  acceptable  (not  including  cost) . 

Question  2.  In  answer  to  the  above  question,  when 
in  relation  to  the  contract  award  date,  were  the 
aspect (s)  evident? 

Question  3.  How  does  the  F-5E,  TIGER  II,  differ 
from  other  F-5  series  aircraft? 


Preceding  pan  blank 


Question  4.  What  system (s) /component (s)  contrib- 
uted to  the  greatest  delays? 

Question  5.  What  information/data  was  available/ 
usable  from  previous  F-5  series  aircraft? 
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Question  6.  Was  this  information/data  used? 
Question  7.  How  was  this  data  used? 

Responses  to  Round  I Questions 

The  answers  of  each  respondent  are  listed  below 
by  individual  interrogated. 

Participant  1.  Participant  number  1 answered  the  questions 
as  follows: 


1.  Question  1, 
identified: 


The  following  aspects  were 


a.  Aircraft  Engines.  The  requirement  for  this 
fighter  aircraft  was  to  utilize  the  same  size  and  weight 
engine  as  the  earlier  model  F-5,  but  increase  the  thrust 
produced  by  the  engines.  It  was  a high  technical  risk 
area  using  new  metals. 

b.  Fire  Control  Radar.  Previous  models  did 
not  have  a radar  set  group.  The  development  of  a new, 
miniaturized  set  to  fit  in  the  structural  dimensions  of 
the  F-5E  nose  section  was  undertaken. 

c . Lead  Computing  Gun  Sight.  This  was  a new 
system  added  onto  the  F-5  series  aircraft.  There  was 
3ome  difficulty  integrating  the  systems  and  obtaining 
an  operationally  acceptable  product.  Previous  models 
used  an  optical  sight. 


2.  Question  2.  These  aspects  became  evident 
about  6 months  to  a year  after  the  contract  award  date. 


3.  Question  3.  Engines  were  changed;  Radar  and 
Lead  Computing  Gun  Sight  were  added. 

4.  Question  4 . Engines  and  Fire  Control  Radar. 

5.  Question  5.  All  previous  T-38  and  F-5  test 
data  available  were  used  to  hasten  the  F-5E  develop- 
ment. 
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6.  Question  6.  Yes. 


7.  Question  7. 
cause  delays. 


Predetermine  areas  which  might 


Participant  2.  Participant  number  2 answered  the  questions 
as  follows: 


1.  Question  1.  The  following  aspects  were 
identified-: 


a.  Aircraft  Engines.  The  aircraft  design 
called  for  new,  more  powerful  engines  in,  basically, 
the  same  dimensional  area  as  the  previous  model. 

b.  Lead  Computing  Sight.  (Same  as  Lead 
Computing  Gun  Sight)  New  addition  to  the  F-5  aeries 
aircraft. 


c.  Radar . (Same  as  Fire  Control  Radar)  New 
addition  to  the  F-a  series  aircraft. 

2.  Question  2.  Problems  were  encountered  with  the 
Radar  and  Lead  Computing  Gun  Sight  about  nine  to  twelve 
months  into  the  Research  and  Development  (R&D)  Program. 
Engine  problems  were  encountered  approximately  one  and 
one-half  years  into  the  R&D  program. 

3.  Question  3.  The  F-5E  differs  from  other  F-5 

series  aircraft  because  it  has:  two  position  nose 

gear;  J85GE-21  engines  in  lieu  of  -13  engines;  Search 
and  Range  Radar  with  a Lead  Computing  Gun  Sight; 
maneuvering  flaps  and  a tail  hook. 

4.  Question  4 . The  system  contributing  to  the 
greatest  delay  was  the  J85GE-21  engines. 

5.  Question  5 . All  data  generated  by  the  earlier 
F-5  model  aircraft  was  available  for  use  and  comparison. 

6.  Question  6.  Yes. 

7.  Question  7.  The  data  was  used  by  the  Source 
Selection  Control  Panel  for  comparative  purposes. 
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Participant  3.  Participant  number  3 answered  the  questions 
as  follows: 

1.  Question  1.  The  following  aspects  were 
identified: 

a.  Boattail  Design.  The  design  was  acceptable 
but  there  was  a fabrication  problem,  using  titanium, 
which  required  more  hours  than  originally  anticipated. 

b.  Environmental  Control  System.  The  original 
system  experienced  bearing  failure  and  did  not  keep 
the  pilot  cool  enough. 

c.  Fire  Control  Radar.  Several  problems 

became  evident  when  the  radar  set  was  built:  it  could 

not  be  used  at  night  or  in  bad  weather;  the  image  was 
"washed  out"  in  the  sun;  poor  Mean  Time  Between  Failures 
(MTBF) . 

d.  Inadequate  Tooling.  Tooling  and  set-up 
were  inadequate  to  meet  the  originally  specified  pro- 
duction rate.  The  problem  was  solved  by  dropping  the 
rate  to  a maximum  of  18  per  month. 

e . Pass  Fatigue  Test  Within  Time  Schedule. 

Some  structural  fatigue  was  experienced  during  testing 
requiring  modification. 

2.  Question  2.  Problems  were  encountered  with  the 
Fire  Control  Radar  and  Environmental  Control  System 
some  time  after  the  start  of  the  test  program  in  1973. 
Fatigue  testing  problems  were  encountered  about  three 
to  four  years  after  the  contract  award  date  (1973-1974) 
when  testing  got  to  8000  aircraft  hours.  Boattail 
design  problems  were  evident  in  the  first  year  to 
eighteen  months;  inadequate  tooling  was  evident  in  the 
first  year. 

3.  Question  3.  The  F-5E  is  d_"i'  isionally  larger 
with  more  powerful  J85GE-21  engines.  It  has  a titanium 
boattail,  maneuvering  flaps,  and  a Fire  Control  Radar/ 
Lead  Computing  Optical  Sight  System  (FCR/LCOSS) . 

4.  Question  4 ♦ The  -21  engines  were  the  greatest 
single  problem;  they  had  not  been  previously  foreseen 
as  a problem  area.  Problems  encountered  were: 

a.  Fifth  and  sixth  stage  compressor  problems. 
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b.  Plameouts  in  certain  portions  of  the 
envelope. 

c.  Variable  Exhaust  Nozzle  problems. 

5.  Question  5.  Technical  data  was  available 
because  of  the  assumed  similarity  between  A/B  and  E 
models.  Only  limited  pricing  data  was  used  because 
prior  aircraft  were  on  a firm  fixed  price  basis  and 
the  F-5E  is  on  an  incentive  contract. 

6.  Question  6.  A great  deal  of  the  technical  data 
was  used  initially.  Pricing  data  was  not  used  exten- 
sively because  commonality  did  not  appear  high  and  data 
had  not  been  collected  in  a consistent  manner. 

7 Question  7.  The  available  historical  data  was 
used  to  compare  learning  curves  for  pricing  and  esti- 
mating purposes.  It  was  also  used  to  qualify  or 
approve  a design  because  it:  was  common  to  the  F-5A/B. 
This  belief  in  commonality  caused  some  problems  since 
commonality  did  not  appear  to  be  as  high  as  originally 
expected. 


Participant  4.  Participant  number  4 answered  the  questions 
as  follows: 

1.  Question  1.  The  following  aspects  were 
identified: 


a.  AGE  1 roblems . Aerospace  Ground  Equipment 
(AGE)  design  followed  hardware  design  thereby  lessening 
lead  time  to  production. 

b.  GFAE  Problems.  Government  Furnished 
Aerospace  Ground  Equipment  (GFAE)  required  long  lead 
time  versus  delivery;  the  rejection  rate  was  also  high. 

c.  Logistic  Problem.  There  was  some  difficulty 
with  late  spare  equipment  and  unserviceable  common  AGE. 

d.  Manufacturing  Problem.  There  was  a high 
learning  curve  with  the  F-5E. 

e.  Pa3s  Fatigue  Test  Within  Time  Schedule. 

There  was  some  uncertainty  that  the  F-5E  would  pass  the 
fatigue  program  on  time.  Retrofitting  was  necessary 

to  extend  service  life  to  4000  hours. 

f.  Pass  Flight  Test  Within  Time  Schedule.  New 
development  projects  added  to  the  F-5E  caused  slippage 
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10.  Manufacturing  Problems ♦ There  was  a high 
learning  curve  involved  in  this  production. 

11.  Pass  Fatigue  Test  Within  Time  Schedule.  Dis- 
covery of  fatigue  areas  required  retrofiting  to  attain 
4000  hour  service  life. 

12.  Pass  Flight  Test  Within  Time  Schedule.  New 
de ve lopment  projects  caused  slippage. 


ROUND  II 


Round  II  of  the  DELPHI  procedure  elicited  proba- 
bility distributions  for  the  aspects  identified  by  the 
participants  in  Round  I.  The  question  used  in  this  round 
included  a summary  of  the  responses  to  Round  I. 


Round  II  Question 


The  following  is  a summary  of  uncertain  aspects  of 
the  F-5E,  TIGER  II,  Development  Program.  We  now  ask  you 
to  assess  the  probability  of  the  outcome  of  an  aspect  being 
in  one  of  the  given  zones.  The  sum  of  the  three  probabil- 
ities for  each  aspect  must  be  1.0.  A brief  statement  of 
your  reasons  is  also  necessary.  Consider  each  aspect 
individually. 


Aspect 

Unacceptably 

Poor 

(Unsatisfactory) 

Acceptable 

(O.K.) 

Unacceptably 
Excellent 
(Too  Good) 

AGE  Problem 

p« 

P= 

P= 

Boattail 

Design 

P“ 

P“ 

P“ 

Environmental 

Control  Sys 

P“ 

P= 

p= 

Engines 

p» 

P= 

P= 

Fire  Control 

Radar 

P= 

P= 

P* 

GFAE  Problem 

P= 

P= 

P=* 

Inadequate 

Tooling 

p» 

P“ 

P= 

1 

i 


i 


i 
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Aspect 


Unacceptably  Unacceptably 

Poor  Acceptable  Excellent 

(Unsatisfactory)  (O.K.)  (Too  Good) 


Lead  Computing 
Gun  Sight  p» 


Logistic 

Problem 


Manufacturing 
Problem  p* 


Pass  Fatigue 
Test  w/i  Time 
Schedule  p» 


Pass  Flight 
Test  w/i  Time 
Schedule  p* 


Responses  to  Round  II  Question 


Table  6 contains  the  response  of  continuing 


participants  to  the  Round  II  question.  It  should  be  noted 


at  this  point  that  only  the  responses  from  3 participants 


are  listed  in  the  Round  II  question  table. 


To  maintain  a semblance  of  continuity  of  informa- 


tion and  train  of  thought,  data  collection  was  designed  to 


cover  a 10  working  day  period  from  start  to  finish.  Long 


periods  of  time  delay  between  sessions  could  possibly  be 


detrimental  to  the  study  in  that  participants  may  not  be 


able  to  reproduce  their  rationale  (4:62).  Due  to  an  unex- 


pected temporary  duty  (TDY)  commitment  across  country 


NOTE;  For  the  purpose  of  this  and  subsequent  questions, 
the  term  acceptability  is  defined  as  the  contract  specifi- 
cation at  development  award  time  plus  or  minus  an  amount 
that  would  have  defined  acceptability  for  you.  Do  not 
attempt  to  quantify  the  range.  Simply  conceive  of  accep- 
table outcomes  as  a range  of  values.  Precise  measurement 
of  parameters  is  not  the  objective  of  this  experiment. 
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Table  6 

Summary  of  Round  II  Responses  to 
DELPHI  Interrogation 


Aspect 

Number 

Participant 

Number 

Unacceptably 

Poor 

Acceptable 

Unacceptably 

Excellent 

1 

1 

0.05 

0.95 

0.00 

2 

0.20 

0.80 

0.00 

3 

0.80 

0.20 

0.00 

2 

1 

0.05 

0.95 

0.00 

2 

0.80 

0.20 

0.00 

3 

0.20 

0.80 

0.00 

3 

1 

0.10 

0.90 

0.00 

2 

0.50 

0.50 

0.00 

3 

0.60 

0.40 

0.00 

4 

1 

0.25 

0.75 

0.00 

2 

0.30 

0.70 

0.00 

3 

0.20 

0.80 

0.00 

5 

1 

0.30 

0.70 

0.00 

2 

0.70 

0.30 

0.00 

3 

0.80 

0.20 

0.00 

6 

1 

0.15 

0.85 

0.00 

2 

0.20 

0.80 

0.00 

3 

0.40 

0.60 

0.00 

7 

1 

0.15 

0.85 

0.00 

2 

0.70 

0.30 

0.00 

3 

0.00 

1.00 

0.00 

8 

1 

0.25 

0.75 

0.00 

2 

0.80 

0.20 

0.00 

3 

0.20 

0.80 

0.00 

9 

1 

0.20 

0.80 

0.00 

2 

0.70 

0.30 

0.00 

3 

0.80 

0.20 

0.00 

10 

1 

0.15 

0.85 

0.00 

2 

0.80 

0.20 

0.00 

3 

0.50 

0.50 

0.00 

A? 

•? 

■i 


Aspect  Participant  Unacceptably  Unacceptably 

Number  Number  Poor  Acceptable  Excellent 


1 

0.10 

0.90 

0.00 

2 

0.60 

0.40 

0.00 

3 

0.00 

1.00 

0.00 

1 

0.10 

0.90 

0.00 

2 

0.80 

0.20 

0.00 

3 

0.00 

1.00 

0.00 

* 

$ 

'J 

* 

I 

1 


I 

X 

J 

I 


! 

1 


J 
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during  the  data  gathering  period,  further  participation  by 
participant  number  4 was  discontinued  in  Rounds  II  and  III. 
All  the  data  accumulated  from  the  four  participants  in 
Round  I were  utilized  in  Rounds  II  and  III  under  the  assump- 
tion that  they  would  not  bias  the  study.  Since  the  DELPHI 
procedure  utilizes  anonymity  of  source  information,  itera- 
tive controlled  feedback,  and  statistical  group  response 
(4:45),  no  adverse  affect  was  anticipated. 

ROUND  III 


Round  III  Question 

The  following  question  was  used  in  Round  III: 

Below  are  listed  the  responses  for  each  aspect  of 
Round  II.  If  you  wish  to  change  any  of  the  responses  in 
light  of  the  summarized  information,  please  do  so  by 
striking  out  the  number  shown  and  enter  another  number. 
Please  state  your  reasons  for  any  change  you  might  make. 

Responses  tc  Round  III  Question 

The  responses  to  the  above  question  are  presented 
in  summary  form  in  Table  7. 

Changes 

There  were  no  changes  in  Round  III  using  the  data 
from  Round  II.  The  probabilities  that  were  obtained  from 
Round  III  were  used  to  measure  the  entropy  required  for 
using  the  Martin  Cost  Model  to  predict  total  program  cost 
for  the  F-5E  development  program  (see  Appendix  B and 
Chapter  III) . Table  8 contains  the  probabilities  from  the 
entropy  data  collection. 
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Table  7 

Summary  of  Round  III  Responses  to  the 
DELPHI  Interrogation 


Probabil ity 

Probability 

of 

Probability 

of 

Aspect 

Participant 

Unacceptably 

of 

Unacceptably 

Number 

Number 

Poor 

Acceptability 

Excellent 

1 

1 

0.05 

0.95 

0.00 

2 

0.20 

0.80 

0.00 

3 

0.80 

0.20 

0.00 

2 

1 

0.05 

0.95 

0.00 

2 

0.80 

0.20 

0.00 

3 

0.20 

0.80 

0.00 

3 

1 

0.10 

0.90 

0.00 

2 

0.50 

0.50 

0.00 

3 

0.60 

0.40 

0.00 

4 

1 

0.25 

0.75 

0.00 

2 

0.30 

0.70 

0.00 

3 

0.20 

0.80 

0.00 

5 

1 

0.30 

0.70 

0.00 

2 

0.70 

0. 30 

0.00 

3 

0.80 

0.20 

0.00 

6 

1 

0.15 

0.85 

0.00 

2 

0.20 

0.80 

0.00 

3 

0.40 

0.60 

0.00 

7 

1 

0.15 

0.85 

0.00 

2 

0.70 

0.30 

0.00 

3 

0.00 

1.00 

0.00 

8 

1 

0.25 

0.75 

0.00 

2 

0.80 

0.20 

0.00 

3 

0.20 

0.80 

0.00 

9 

1 

0.20 

0.80 

0.00 

2 

0.70 

0.30 

0.00 

3 

0.80 

0.20 

0.00 

10 

1 

0.15 

0.85 

0.00 

2 

0.80 

0.20 

0.00 

3 

0.50 

0.50 

0.00 
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Table  7 (continued) 


Aspect 

Number 

Participant 

Number 

Probability 

of 

Unacceptably 

Poor 

Probability 

of 

Acceptability 

Probability 

of 

Unacceptably 

Excellent 

11 

1 

0.10 

0.90 

0.00 

2 

0.60 

0.40 

0.00 

3 

0.00 

1.00 

0.00 

12 

1 

0.10 

0.90 

0.00 

2 

0.80 

0.20 

0.00 

3 

0.00 

1.00 

0.00 

f: 

t 

| 

s 

< 

\ 

\ 


t 
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Table  8 


Entropy  Data  Collection  Results 


Outcome  Probabilities 

Unacceptably 

Unacceptably 

Aspect 

Poor 

Acceptable 

Excellent 

AGE  Problem 

.350 

.650 

0.00 

Boattail 

Design 

.350 

.650 

0.00 

Environmental 
Control  Sys 

.400 

.600 

0.00 

Engines 

.250 

.750 

0.00 

Fire  Control 
Radar 

.600 

.400 

0.00 

GFAE  Problem 

.250 

.750 

0.00 

Inadequate 

Tooling 

.280 

.720 

0.00 

Lead  Computing 
Gun  Sight 

.417 

.583 

0.00 

Logistic 

Problem 

.560 

.440 

0.00 

Manufacturing 

Problem 

.483 

.517 

0.00 

Pass  Fatigue 
Test  w/i  Time 

Schedule 

.233 

.767 

0.00 

Pass  Flight 
Test  w/i  Time 
Schedule 

.300 

.700 

i 

0.00 
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APPENDIX  B 

COMPUTATION  OF  F-5E 
PROGRAM  ENTROPY 

The  results  of  the  DELPHI  data  collection  with 
respect  to  the  F-5E,  TIGER  II,  development  program,  are 
summarized  in  Chapter  III  (p.  81)  and  shown  in  Appendix  A. 
Appendix  B will  explain  the  details  of  computing  entropy 
given  the  conditions  specified  by  the  DELPHI  results. 

ENTROPY  CALCULATION  FORMULA 

The  formula  for  determining  entropy  is: 

m 

Pi  log  Pi 

H _ i«l 

«max  " lQ9  m 

number  of  possible  outcomes 

probability  of  i*-*1  outcome 

unadjusted  entropy  value 

maximum  entropy 

expression  of  relative  disorder  or 
entropy 

the  F-5E  DELPHI  results,  a total  of 
12  program  aspects,  each  with  three  possible  outcomes, 
were  identified.  The  specific  outcome  of  each  of  the  12 
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Preceding  pate  blank 


wnere : ra  = 

Pi  * 
H = 

^max 
h = 

In  the  case  of 





aspects  described  a program  outcome.  Assuming  independence 


among  aspects,  the  probability  of  a program  outcome  is 


evaluated  as  the  product  of  each  of  the  12  specific  outcome 


probabilities  (16:100). 


The  number  of  possible  outcomes  is  derived  from 


each  program  as  the  number  of  possible  combinations  of  two 


things  (aspect  outcomes)  taken  one  at  a time  (specific 


aspect  outcome)  (16:101).  A program  outcome  is  a sequence 


of  twelve  aspect  outcomes,  and  may  be  evaluated  as  follows: 


Number  of  aspect  outcomes  * (j)  (16:101) 


Sequence  of  twelve  aspects 


<?)12 


( illy) * 2^2  = 4096  program  outcomes. 


During  the  data  gathering  process,  the  participants 


expressed  their  probabilities  as  either  Unacceptably  Poor 


or  Acceptable.  The  category  labeled  Unacceptably  Excel- 


lent was  eliminated  by  the  participants  as  nonexistent  and 


therefore  was  eliminated  as  a possible  outcome  for  our 


computational  formulation. 


ENTROPY  COMPUTER  PROGRAM 


A modified  version  of  Glover-Lenz  computer  program 


in  FORTRAN  IV  programming  language  was  used  on  the  Air 


Force  Logistics  Command  CREATE  computer  system.  The  pro- 


gram listing  is  shown  in  Figure  11  with  Variable  Names  and 


Definitions  in  Table  9. 
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010  DIMENSION  PMAT(12,2) 

015  DOUBLE  PRECISION  PEVENT, P SUM, PLOG, HSUM, HMAX,H,NE 
P2P  PRINT, "INPUT  THE  PROBABILITY  MATRIX  ROW-WISE." 

P3p  READ, ( (PMAT (I, J) , J-l, 2) , 1*1,12) 

P35  NE  - p 
P4p  EVNTNO -0 
p5p  PSUM-0 
P6P  HSUM -0 
P7p  DO  10  11-1,2 
080  DO  10  12-1,2 
090  DO  10  13-1,2 
100  DO  10  14-1,2 
110  DO  10  15-1,2 
120  DO  10  16-1,2 
130  DO  10  17-1,2 
140  DO  10  18-1,2 
150  DO  10  19-1,2 
160  DO  10  110-1,2 
170  DO  10  111-1,2 
180  DO  10  112-1,2 

190  PEVENT— PMAT ( 1 , I 1 ) *PMAT (2 ,12) *PMAT (3,13) *PMAT (4 ,14) *PMAT (5 , 15) * 
2004PMAT (6 , 16) *PMAT (7 , 17) *PMAT (8 , 18) *PMAT (9 , 19) *PMAT (10,110) * 
2104PMAT (11 , 111) *PMAT (12 , 112) 

220  PSUM-PSUM  + PEVENT 

230  PLOG«PEVENT*ALOG10 (PEVENT) 

240  HSUM-HSUM  + PLOG 
250  EVNTNO— EVNTNO  + 1. 

260  10  CONTINUE 
270  HMAX-ALOG10 (EVNTNO) 

280  HSUM-HSUM* (-1.0) 

290  H-HSUM/HMAX 
295  NE  * 1- (H) 

300  PRINT, "H« 

305  PRINT," NE= 

310  PRINT, "HSUM- 
320  PRINT, "HMAX- 
330  PRINT, "EVNTNO- 
340  PRINT, "PSUM- 
350  STOP 
360  END 


»« 

" ,NE 
" , HSUM 
" ,HMAX 
" , EVNTNO 
" ,PSUM 


Figure  11 

Entropy  Computation  Program 
Coding 


1 

i 

i 


i 

i 

1 
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Table  9 

Variable  Names  and  Meaning 


Variable  Name 

Meaning 

PMAT 

A matrix  of  program  feature  outcome 
probabilities. 

I,J 

Indices  for  the  PMAT  matrix. 

EVNTNO 

Number  of  program  outcomes. 

PSUM 

Sum  of  the  program  outcome  proba- 
bilities. 

HSUM 

Sum  of  the  products  pi  log  p^. 

PEVENT 

Probability  of  a program  outcome. 

PLOG 

Product  of  PEVENT  and  log^Q  PEVENT. 

UMAX 

The  maximum  entropy  (log^0  EVNTNO) . 

H 

The  expression  of  relative  entropy. 

NE 

The  degree  of  order  in  the  system, 
1 - relative  entropy. 

11, . . . ,19 

DO  Loop  counters. 

J'.tt  i .1 
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To  insure  the  reliability  of  the  Glover-Lenz 
Entropy  Computation  Program,  a small  version,  as  test,  was 
manually  calculated.  The  data  and  results  are  in  Table  10. 
The  computer  program  for  these  calculations  is  listed  in 
Figure  12  with  results  in  Figure  13. 


Table  10 

Data  and  Results  of  2 x 2 
Manual  Calculation 


Outcome 


Aspect 


Poor 


Acceptable 


1 .5 

2 .3 

The  total  number  of  events  possible  are: 

4 = ro  (33:121-123) 


.5 

.7 


(?) 2 


, 21 

(mr) 


(.5)  (.3)  * .15 
(.5) (.7)  * .35 


(.3)  (.5)  = .15 
(.7) (.5)  * .35 


H 


sum 


-E- 


log  pi 


-(.15)  (-.823909)  = (.1235863)  (2)  = .2471726 
-(.35)  (-.455932)  « (. 1595762) (2)  = .3191524 


H 


sum 


.5663250 


H * log  m « log  4 = .60206 
m&x 


H 


sum 


.5663250 


Hma) 


.60206 


.9406454 


I 

I 

1 


i 
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010  DIMENSION  PMAT [2,2) 

020  PRINT, "INPUT  NOW" 

030  READ, ((PMAT (I, J) ,J»1,2) ,1-1,2) 

040  EVNTNO-0 

050  PSUM-0 

060  HSUM-0 

070  DO  10  11-1,2 

080  DO  10  12-1,2 

090  PEVENT— PMAT (1,11) *PMAT(2 ,12) 

100  PSUM-PSUM  + PEVENT 

110  PLOG-PEVENT*ALOG10 (PEVENT) 

120  HSUM-HSUM+PLOG 
130  EVNTNO— EVNTNO  + 1 
140  10  CONTINUE 
150  HMAX-ALOG10 (EVNTNO) 

160  HSUM-HSUM*  (**1.0) 


170  H-HSUM/HMAX 
180  PRINT, "H-  " ,H 

190  PRINT, "HMAX-  " ,HMAX 

200  PRINT, "HSUM-  ",HSUM 

210  PRINT, "EVNTNO*  ", EVNTNO 

220  PRINT, "PSUM-  ",PSUM 

230  STOP 
240  END 


Figure  12 

Modified  2x2  Entropy 
Computer  Program 


INPUT  NOW 
*.5, .5 
*.3, .7 

H—  0.94064546E  00 

HMAX-  0 . 60205999E  00 

HSUM-  0. 56632499E  00 

EVNTNO-  0.40000000E  01 

PSUM-  0.10000000E  02 


Figure  13 

Data  and  Results  of  Modified  2x2 
Entropy  Computer  Program 
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A second  set  of  calculations,  a little  more  com- 
plex than  the  first,  were  manually  calculated  with  the 
data  and  results  listed  in  Table  11.  The  results  utilizing 
the  computer  program  listed  in  Figure  14  are  in  Figure  15. 

Table  11 

Data  and  Results  of  3 x 2 
Manual  Calculation 


Outcome 


Aspect 


Poor 


Acceptable 


The  total  number  of  possible  events  are: 

(1)3  * (lflT)3  = 23  * 8 - m (33:121-123) 


pi 

(.5) (.3) (.2)  * .03 

(.5) (.3) (.2)  = 

Pi 

.03 

(.5) (.3) (.8)  * .12 

(.5)  (.3)  (.8)  = 

.12 

(.5) (.2) (.7)  = .07 

(.5)  (.2) (.7)  » 

.07 

(.5) (.7) (.8)  - .28 

(.5)  (.7) (.8)  - 

.28 

H - -Vp. 

sum  / i 


log  p. 


« -(.03)  (-1.522878745)  = (.  0456863623)  (2)  = .0913727246 

* -(.12)  (~  .920818754)  = ( .1104982505)  (2)  = .220996501 

» -(.07)  (-1.15490196  ) = (. 08084 31372)  (2)  = .1616862744 

= - ( . 28) {-  .5528419687)  = (. 1547957512) (2) = .3095915024 


H = .7836470024 
sum 


log  m - log  8 = .903089987 


h = Hsum/Umax  = * 7836470024/. 903089987  = .8677396646 
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10  DIMENSION  PMAT (3,2) 

20  PRINT, "INPUT  THE  PROBABILITY  MATRIX  ROW-WIRE." 
30  READ, ((PMAT(I,J) ,J*1,2) ,1*1,3) 

40  EVNTNO- 0 
50  PSUM-0 
60  HSUM-0 
70  DO  10  11-1,2 
80  DO  10  12-1,2 
90  DO  10  13-1,2 

100  PEVENT-PMAT (1, I 1 ) *PMAT (2 ,12) *PMAT (3 , 13} 

110  PSUM-PSUM+PEVENT 

120  PLOG-PEVENT*ALOG10 (PEVENT) 

130  HSUM-HSUM+PLOG 
140  EVNTNO— EVNTNO+ 1 
150  10  CONTINUE 
160  HMAX-ALOG10 (EVNTNO) 

170  HSUM-HSUM* (-1.0) 

180  H-HSUM/HMAX 

190  PRINT, "H=  " ,H,"  HSUM-  ",HSUM 

200  PRINT, "HMAX-  " , HMAX , " EVNTNO-  ", EVNTNO 

210  PRINT, "PSUM-  " ,PSUM 

220  STOP 

230  END 


Figure  14 

Modified  3x2  Entropy 
Computer  Program 


INPUT  THE  PROBABILITY  MATRIX  ROW-WISE. 

=. 5 , .5, .3, .7, .2, .8 

H=  0. 86773967E  00  HSUM=  0.78364701E  00 
HMAX-  0 . 90308999E  00  EVNTNO=  0-80000000E  01 

PSUM-  0.10000000E  01 


Figure  .15 

Data  and  Results  of  3 x 2 
Computer  Calculation 
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10  DIMENSION  PMAT (3,2) 

20  PRINT, "INPUT  THE  PROBABILITY  MATRIX  ROW-WISE." 
30  READ, ( (PMAT(I,J) ,J-1, 2) ,1-1,3) 

40  EVNTNO-0 
50  PSUM-0 
60  HSUM-0 
70  DO  10  11-1,2 
80  DO  10  12-1,2 
90  DO  10  13-1,2 

100  PEVENT-PMAT (1,11) *PMAT (2,12) *PMAT (3,13) 

110  PSUM-PSUM+PEVENT 

120  PLOG-PEVENT*ALOG10 (PEVENT) 

130  HSUM-HSUM+PLOG 
140  EVNTNO-EVNTNO+ 1 
150  10  CONTINUE 
160  HMAX-ALOG10 (EVNTNO) 

170  HSUM-HSUM* (-1.0) 

180  H-HSUM/HMAX 

190  PRINT, "H=  " , H , " HSUM-  " ,HSUM 

200  PRINT, "HMAX-  " , HMAX , " EVNTNO-  ", EVNTNO 

210  PRINT, "PSUM-  " , PSUM 

220  STOP 

230  END 


Figure  14 

Modified  3x2  Entropy 
Computer  Program 


INPUT  THE  PROBABILITY  MATRIX  ROW-WISE. 

H=  # 0.86773967E%0  HSUM-  0.78364701E  00 

HMAX=  0 . 90308999E  00  EVNTNO-  0.80000000E  01 

PSUM-  0.10000000E  01 


Figure  .15 

Data  and  Results  of  3 x 2 
Computer  Calculation 
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It  is  evident  that  the  mathematical  computations 
are  compatible  via  manual  and  computer  methods.  The  final 
results  for  the  calculation  of  entropy  are  identical  and 
the  computer  program  is  correct  and  functional. 

The  input  data  for  the  program  is  the  series  of  24 
probability  values  from  the  DELPHI  data  collection,  entered 
sequenticlly  for  each  program  feature.  The  program  com- 
putes each  F-5E  program  outcome  probability,  evaluates  its 
logarithm  to  the  base  10  and  computes  total  entropy,  maxi- 
mum entropy,  and  relative  entropy.  The  input  and  output 
for  the  computations  in  Chapter  III  are  shown  in  Figure  16; 
the  input  values  were  taken  from  Table  8,  Appendix  A. 

These  values  represent  an  average  value,  by  category,  for 
each  aspect. 
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INPUT  THE  PROBABILITY  MATRIX  ROW-WISE 

*.35, .65 

*.35, .65 

*.  4 , . 6 

*.25, .75 

*.  6, . 4 

*.25, .75 


*. 28 , . 72 
= .417, .583 
*.56, .44 
=.483, .517 
=.233, .767 
*.3, .7 
H= 

NE= 

HSUM* 

HMAX= 

EVNTNO* 

PSUM* 


0. 91011028 1874 6972 30D  00 
0.898897 18 12 5 3027707D-01 
0. 328764593543351697D  01 
0. 3612 3599 5 34129 2 780D  01 
0.40960000E  04 
0.99999999 3^856663970  00 


Figure  16 

F-5E  Entropy  Computation 
Input  and  Output 
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STATISTICAL  TEST  OF  THE 
MODEL 


This  appendix  presents  the  calculations  used  to 


support  the  test  of  the  Martin  Cost  Model  as  a predictor 


of  program  development  cost.  The  F-test  for  homogeneity 


of  variances  was  used  to  compare  the  following  variances: 


Variance  of  Errors — Variance  of  errors  measured 
from  the  observed  quarterly  cumulative  costs  to  a 
least-squares  line  fitted  to  the  actual  cost  obser- 
vations. 


Variance  of  Deviations— -Variance  of  the  deviations 
measured  from  the  observed  quarterly  cumulative  costs 
to  a predicted  cost  line  based  on  the  Martin  Cost  Model 
prediction  for  total  program  cost. 


After  the  variances  were  calculated,  they  were  compared  to 


determine  if  they  were  from  the  same  population  variance. 


The  F-test  was  used  for  this  comparison. 


CALCULATION  OF  VARIANCES 


Least-Squares  Line  of  Best  Fit 


Before  the  variance  of  the  errors  could  be  ealeu 


lated,  a least-squares  line  had  to  be  fitted  through  the 


observed  cost  data  and  the  errors  of  the  cost  points 


measured  from  the  line.  Using  the  assumption  of  linear 
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cost  and  applying  the  method  of  least-squares  to  the 
observed  cost  data  listed  in  Table  12,  a least-squares  line 
of  best  fit  was  computed.  These  cumulative  cost  figures 
for  each  quarterly  time  period  were  obtained  through  the 
F-5  International  Fighter  SPO’s  Financial  Management  Sec- 
tion and  were  based  upon  the  Contract  Funds  Status  Report 
(CFSR)  for  the  F-5E.  The  CFSR  is  a report,  published 
quarterly  by  the  contractor,  reflecting  cumulative  actual 
cost  as  of  the  calendar  quarter. 

The  following  steps  were  used  to  construct  the 
equation  for  the  least-squares  line. 


Slope  Computation.  The  first  step  of  the  process  was  to 
calculate  the  slope  of  the  line  as  follows  (27:368-370): 


m 


14  14  14 


14 

E 

ti=i  • E **  E °i 

i-1 

i-1  i-1 

14 

14 

14 

E 

< - (E*.)2 

i-1 

' i-1  ' 

Thus,  m « 6.8475. 


Intercept  Computation.  The  next  step  was  to  calculate  the 
vertical  intercept  (y)  for  the  line  (27:576). 

y * c - mt 

where:  t * the  mean  number  of  quarters 

m = slope  of  the  line 


l 

* 

i 


i 

j 

i 

i 


A 


c » the  mean  of  the  observed  costs 
y ■ the  vertical  intercept  of  the  line 
Thus,  y * 8.7713. 

Therefore , the  least-squares  line  representing 

the  quarterly  cumulative  cost  data  is: 

C.  » 8.7713  + 6. 8475t . 
l i 

where:  ■ the  ith  quarter  during  the  develop- 

ment program,  and 

Ci  * cumulative  cost  estimated  at  the 
end  of  the  !*■"  quarter. 

This  equation  was  used  to  measure  errors  of  the  observed 
cost  points  from  the  least-squares  line. 


Variance  of  the  Errors.  The  errors  of  the  observed  cost 
data  from  the  least-squares  line  were  calculated  using  the 


following  equation: 


°i  - ci 


where:  Cj^  * observed  cost  for  the  ith  quarter 

* cost  calculated  for  the  ith 
quarter  using  the  least-squares 
equation 

e^  « error  for  the  ifc^  cost  point 

measured  from  the  lea3t~squares 
line 

The  results  of  these  calculations  are  presented  in 
Table  13. 


Calculation  of  the  Variance.  Following  the  measurement  of 
the  errors  of  the  observed  cost  points  from  the  least- 
squares  line,  the  variance  of  errors  was  calculated.  The 
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Table  13 

Errors  About  Least-Squares 
Line 


i 

ci 

Ci 

ei 

1 

4.510 

15.6188 

-11.1088 

2 

10.573 

22.4663 

-11.8933 

3 

20.833 

29.3138 

-8.4808 

4 

38.299 

36.1613 

+2.1377 

5 

48.554 

43.0088 

+5.5452 

6 

59.140 

49.8563 

+9.2837 

7 

69.733 

56.7038 

+13.0292 

8 

77.609 

63.5513 

+14.0577 

9 

81.081 

70.3988 

+12.8252 

10 

83.224 

77.2463 

+5.9777 

11 

84.399 

84.0938 

+0.3052 

12 

86.253 

90.9413 

-4.6883 

13 

88.256 

97.7888 

-9.5328 

14 

89.322 

104.6363 

-15.3143 

variance  was  calculated  using  the  following  equation 
(27:573): 


n 


c2  » i*! 

e n - 1 

where:  n » number  of  quarters 

e^  * error  of  the  ifc^  cost  point 

measured  from  the  least-squares 
line 

e * mean  of  the  errors 
2 

Sg  *>  sample  variance  of  the  errors 

The  result  of  this  calculation  was  * 105.6178.  This 
variance  was  later  compared  with  the  variance  of  deviations 
in  the  F-test. 

Variance  of  the  Deviations 

Before  determination  of  the  variance  of  the  devia- 
tions, a cost  prediction  line  using  the  Martin  Cost  Model 
prediction  and  the  deviations  of  observed  cost  data  points 
measured  from  this  line  was  determined. 

Predicted  Cost  Line.  The  calculations  for  measuring 
entropy  for  use  in  predicting  the  total  program  cost  with 
the  Martin  Cost  Model  are  found  in  Appendices  A and  B. 

The  calculated  total  program  cost  was: 

CI  83.635  83.635 

CE  * 1 - h “ 1 - .91  = T09 


Therefore,  the  total  program  cost  predicted  by  the  model 
was  $929,278  million. 

Using  an  assumption  of  linear  cost,  a line  was 
drawn  from  the  origin  of  the  time-cost  plane  to  the  pre- 
dicted cost  point,  Cg  * 929.278  plotted  at  T-^.  The 
point-slope  method  was  used  to  determine  the  equation  for 
this  line  as  follows  (16:111): 


cE  - 0 

T14  - 0 


929.278 

14 


wm 


where:  m * slope  of  the  line 

Therefore,  the  equation  for  the  predicted  cost 


line  is: 


CEi  » 66.377^ 


where:  T^  * ith  quarter 

Cjj  * Martin  Cost  Model  cost  forecast 
* at  quarter  T^ 


Calculation  of  the  Deviations.  The  following  equation  was 
used  to  calculate  the  deviations  of  observed  cost  data 
from  the  predicted  cost  line: 

di  - ci  - cEi 

where:  c^  * observed  cost  for  i***1  quarter 

CE.  = calculated  cost  for  ith  quarter 
1 using  predicted  cost  line  equation 

d^  = deviation  of  ifc^  cost  point 

measured  from  the  predicted  cost 
line 


The  results  of  these  calculations  are  presented  in  Table  14. 


Table  14 

Deviations  About  The  Predicted 
Cost  Line 
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Calculation  of  the  Variance.  After  measuring  the  devia- 
tions of  observed  costs  from  the  predicted  cost  line,  the 
variance  of  the  deviations  was  calculated.  The  variance 
was  determined  using  the  following  equation  (27:573): 


n 


where:  n ■ number  of  quarters 

d^  » deviation  of  ith  cost  point 

measured  from  predicted  cost  line 

2 

S,  * sample  variance  of  the  deviations 
a 

The  result  of  this  calculation  was  S^j  » 269125.51.  The 
variance  calculation  was  compared  with  the  variance  of  the 
errors  using  the  F-test  presented  below. 

Statistical  Test 

Statement  of  the  Hypotheses.  The  statement  of  the  hypoth- 
eses in  the  statistical  test  was: 

Null  Hypothesis  (H0) : o|  * 

Alternate  „ . 

Hypothesis  (H^) : °e  ^ °d 

As  stated,  the  hypotheses  represent  a two-tail  test  com- 
paring variances  (16:114). 

Computation  of  Calculated  F.  The  following  equation  was 
used  to  determine  the  value  for  calculated  F,  denoted  by 
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Fc,  with  13  degrees  of  freedom  (16:114): 

„ Larger  Variance 

smaller  Variance 


Sd  269125.51 
s2  * 105.6178 


2548.1075 


Statement  of  the  Decision  Rule.  The  following  decision 
rule  was  used  to  test  the  homogeneity  of  the  two  variances 
(16:115) . 

If  Fp(13,13)  is  greater  than  the  critical  F, 
denoted  F*,  then  HQ  can  be  rejected. 

If  Fc(13,13)  is  less  than  or  equal  to  F* , 
then  H0  cannot  be  rejected. 

F*  is  the  critical  value  determined  from  a book  of  standard 
statistical  tables. 

Determination  of  F*.  F*  for  13  degrees  of  freedom  was 
calculated  at  the  one,  five,  and  ten  percent  significance 
level  using  a book  of  standard  statistical  tables 
(27:621-628).  The  results  of  the  calculations  are  pre- 
sented in  Table  15. 

Table  15 

Critical  Values  of  F 


Significance 

Level 


Critical 
Value  (F*) 


* 


Decision*  Since  F (13,13)  * 2548.1075  is  greater  than  F* 

w 

at  the  one,  five,  and  ten  percent  significance  levels,  HQ 

could  be  rejected.  The  F-test  for  homogeneity  of  variances 

2 2 

indicated  that  there  was  significance  between  Se  and  S^. 
This  result  implies  that  both  variances  are  not  estimates 
cf  a (the  population  variance) , and  are  not  sample 
variances  belonging  to  the  same  population.  Therefore,  the 
research  hypothesis  of  this  study  can  be  rejected. 


APPENDIX  D 

COMPARISON  WITH  TK£ 

GLOVER-LENZ  STUDY 

The  methodology  used  in  the  Glover-Lenz  study 
formed  the  basis  for  the  current  study.  However/  the 
authors  of  the  current  study  could  not  confirm  the  relative 
entropy  value  of  .686  presented  in  the  Glover-Lenz  study  as 
stemming  from  Figure  14  (p.  104)/  Table  3 (p.50) , Table  6 
(p.  95),  or  Table  8 (p.98)  of  the  Glover-Lenz  thesis. 

These  tables  and  figure  are  presented  in  this  appendix  and 
consist  of  the  following  data: 

Figure  17  Glover  and  Lenz  Figure  14 

Entropy  Computation  Input  and  Output. 

Table  17  Glover  and  Lenz  Table  3 

Entropy  Data  Collection  Results 

Table  18  Glover  and  Lenz  Table  8 

Summary  of  Round  III  Responses  to 
DELPHI  Interrogation 

Table  20  Glover  and  Lens  Table  6 

Summary  of  Round  II  Responses  to 
DELPHI  Interrogation 

DISCREPANCIES  IN  GLOVER-LENZ  STUDY 


The  data  presented  in  these  tables  and  figure 
reportedly  formed  the  basis  for  calculating  relative 


; 
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entropy  leading  to  the  subsequent  statistical  test  of  the 
Martin  Cost  Model  which  supported  their  research  hypothesis. 
Errors  in  the  data  presentation  by  Glover  and  Lenz,  appar- 
ently typographical,  have  resulted  in  the  currcn*-  authors' 
inability  to  confirm  the  specific  values  presented  in  the 
earlier  study  and  presented  in  Table  16. 

Figure  17  reflects  the  input-output  data  displayed 
in  the  Glover-Lenz  Figure  14.  The  input  data  was  used  in 
their  computer  program  for  calculating  relative  entropy. 
This  computer  program  is  the  same  one  used  in  the  current 
study  and  presented  in  Appendix  B.  After  verifying  the 
accuracy  of  the  program,  the  authors  of  the  current  study 
found  that  the  input  data  presented  in  Glover-Lenz  Figure 
14  did  not  produce  the  output  data  as  presented  in  that 
Figure  14.  Using  their  input  data,  a relative  entropy  (h) 
of  .70575420  was  output  utilizing  the  given  computer  pro- 
gram. It  was  found,  through  repeated  cross  referencing, 
that  the  values  for  the  Unacceptably  Poor  category  of  the 
Radar  Cross  Section  feature  and  Missile  Speed  feature  were 
not  .2175  and  .2250  as  reflected  in  Glover-Lenz  Table  3, 
but  actually  .2125  and  .255  respectively.  The  FB-111 
Interface,  Unacceptably  Excellent  category  was  .05  in 
Glover-Le?iz  Figure  14  and  .1375  in  Glover-Lenz  Table  3. 

Table  17  reflects  the  data  presented  in  Glover-Lenz 
Table  3.  fable  18  presents  the  arithmetic  mean  of  each 
feature  category  or  probability  outcome  category  for  each 
area  of  uncertainty  in  the  SRAM  development  as  presented 


Table  16 

Relative  Entropy  Summarization  from 
Glover-Lenz  Study  (16:53) 


m « n * 19.683  * number  of  outcomes 
19.683 

H * - log  p^  * 2.9464 


Hmax  « log  19C83  " 4.2941 

h * | * 4!'l'94i  “ °*686  * relative  entropy 

max 

Model  Final  Cost  Prediction  (16:56,107) 


„ _ _ 143.3 

CB  F^h  r~'686 


$456.4  million 


CE  * estimated  final  cost;  Cj  * initial  cost 
estimate  (in  millions) 

CA  * $439.1  million  * actual  final  cost 


F-Tast  (16:54,58-59) 


]e  812.36 
2 * 779.06 


« 1.04 


where  21  quarterly  time  periods  were  involved. 


Findings  (16:115) 


Fc  < F*  with  20  degrees  of  freedom  at  the  .01,  .05, 
and  .10  level  of  significance.  HQ  ccuid  not  be 
rejected. 
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.325, *625# .05 
. 1375# .725, .1375 
.475, .475, .05 
.2125, .700, .0875 
.1375,. 825,. 0375 
.475, .45, .075 
.225, .7125, .0625 
.255, .7125, .0625 
.0875, .825, .0875 


H* 

0.68614852E 

00 

HSUM- 

0.29463844E 

01 

HMAX” 

0.42940913E 

01 

EVNTNO- 

0.19683000E 

05 

PSUM- 

0.99999975E 

00 

Figure  17 

Entropy  Computation  Input  and  Output, 
Glover-Lenz  Figure  14 
(16:104) 


J 


Table  17 


Entropy  Data  Collection  Results, 
Glover-Lenz  Table  3 
(16:50) 


Unacceptably 

Unacceptably 

Feature 

Poor 

Acceptable 

Excellent 

1 

.3250 

.6250 

.0500 

2 

.1375 

.7250 

.1375 

3 

.4750 

.4750 

.1375 

4 

.2175 

.7000 

.0875 

5 

.1375 

.8250 

.0375 

6 

.4750 

.4500 

.0750 

7 

.2250 

.7125 

.0625 

8 

.2250 

.7125 

.0625 

9 

.0875 

.8250 

.0875 

Table  18 

Summary  of  Round  III  Responses 

to 

DELPHI  Interrogation, 

Glover-Lenz  Table  8 

(16:98) 

Unacceptably 

Unacceptably 

Feature 

Poor 

Acceptable 

Excellent 

1 

.2750 

.6500 

.0750 

2 

.1875 

.7000 

.1125 

3 

.4250 

.5000 

.0750 

4 

.2125 

.7000 

.0875 

5 

.1375 

,8250 

.0375 

6 

.4750 

.4500 

.0750 

7 

.2250 

.7125 

.0625 

8 

.2250 

,7125 

.0625 

9 

.0875 

.8250 

.0875 
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in  Glover-Lenz  Table  8.  It  was  the  data  from  this  latter 
table  which  was  to  be  input  data  for  calculating  relative 
entropy  by  way  of  the  computer  program  (16:101).  Aside 
from  discrepancies  in  input  data  of  Glover-Lenz  Figure  14, 
which  should  have  the  same  data  as  their  Table  3,  the 
arithmetic  means,  as  presented  in  the  Glover-Lenz  Table  3, 
are  discrepant  with  their  Table  8.  Figure  14,  Table  3,  and 
Table  8 of  the  Glover-Lenz  study  should  have  the  same  mean 
value.  The  mean  values  of  their  Table  8 have  been  repro- 
duced and  are  reflected  in  Table  18  of  the  current  study. 
Their  Table  8 represented  the  third  and  final  round  of  the 
DELPHI  interrogation  and  formed  the  basis  for  the  values 
in  their  Table  3 (16:97).  The  discrepancies  between  their 
Table  3 and  Table  8 are  presented  in  summary  form  in 
Table  19.  It  should  also  be  noted  that  the  sum  of  the 
probabilities  for  the  FB-111  Interface,  as  presented  in 
their  Table  3,  is  greater  than  1.0. 

In  attempting  to  determine  the  origin  of  the 
discrepancies  in  Table  18,  the  authors  of  the  current  study 
found  that  the  values  for  discrepancy  number  one  through 
eight  apparently  came  from  Glover-Lenz  Table  6 (16:95), 
Summary  of  Round  II  Responses  to  DELPHI  Interrogation,  as 
shown  in  Table  20.  The  origin  of  discrepancy  number  nine 
and  ten  could  not  be  determined. 

Although  a relative  entropy  (h)  of  .686  and  an  F 
calculated  *FC)  of  1.04  were  reported  in  the  Glover-Lenz 
study  as  leading  to  acceptance  of  the  null  hypothesis  and 
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Table  19 


Discrepancies  Between  Table  3 and  8 
of  Glover-Lenz  Study 


Discrepancy 

Number 

Feature 

Probability 

Category 

Table  3 
Value 

Table  8 
Value 

1 

Rocket 

Motor 

Performance 

U.P. 

.3250 

.2750 

2 

A 

.6250 

.6500 

3 

U.E. 

.0500 

.0750 

4 

Missile 

Accuracy 

U.P. 

.1375 

.1875 

5 

A 

.7250 

.7000 

6 

U.E. 

.1375 

.1125 

7 

FB-111 

Interface 

U.P. 

.4750 

.4250 

8 

A 

.4750 

.5000 

9 

U.E. 

.1375 

.0750 

10 

Radar 

Cross 

Section 

U.P. 

.2175 

.2125 

Probability  Category  Code: 

U.P.  « Unacceptably  Poor 

A * Acceptable 

U.E.  * Unacceptably  Excellent 
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Table  20 

Sumary  of  Averages  of  Round  II  Responses  \ 

to  DELPHI  Interrogation,  \ 

Glover-Lenz  Table  6 

(16:96)  \ 


Aspect 

Number 

Unacceptably 

Poor 

Acceptable 

Unacceptably 

Excellent 

1 

.3250 

.6250 

.0500 

2 

.1375 

.7250 

.1375 

3 

.4750 

.4750 

.0500 

4 

.2125 

.7000 

.0875 

5 

.1375 

.8250 

.0375 

6 

.4125 

.5125 

.0750 

7 

.2250 

.7125 

.0625 

8 

.2125 

.7250 

.0625 

9 

.0875 

.8250 

.0750 

i 

i 

l 

I 

t 

i 


t ; 


i 


I 


I 

I 


L 
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support  of  their  research  hypothesis,  these  values  could 
not  be  duplicated  using  the  data  provided.  The  discrepancy 
between  each  individual  probability  value  was  less  than  .10 
and  did  not  appear  to  represent  a gross  error.  Keeping  in 
mind  there  were  no  further  data  available  to  verify  the 
validity  of  Round  II  and  Round  III  data  of  the  Glover-Lenz 
study,  a range  of  possible  entropy  and  Fc  values  were  pre- 
pared using  the  referenced  tables  and  figure  and  are 
reflected  in  Table  21.  It  was  necessary,  however,  to  make 
a correction  to  the  formula  for  Calculation  of  Variance  of 
the  errors  of  the  observed  points  (16:109)  and  for  the 
variance  of  the  deviations  (16:112)  before  continuing  the 
comparison.  Table  22  presents  the  original  formula  used 
by  Glover  and  Lenz  (16)  and  the  corrected  formula  (27:573) 
necessary  for  the  computation  of  the  variances.  If  it  can 
be  assumed  that  the  probability  assessments  of  the  partici- 
pants for  the  second  and  third  round  of  the  DELPHI  interro- 
gations are  either  correct  as  presented  in  the  Glover-Lenz 
study  or  contain  very  few  and  only  slight  differences  from 
the  actual  data  that  was  collected  (similar  to  those 
differences  cited  in  Appendix  D) , then  each  table  and 
figure  that  is  pertinent  to  relative  entropy  and  subsequent 
test  calculations  may  be  considered  separately  and  compari- 
sons made  to  gain  insight  into  whether  or  not  the  finding 
of  the  Glover-Lenz  study  is  accurate.  Such  assumptions 
were  made  by  the  authors  of  the  current  study.  Each  of 
three  tables  and  a figure  that  were  deemed  as  being 
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Table  21  (continued) 
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Data 

Sources 

Values 

Figure  14 
Data 

Id?  - 31428.699 

3 - 25.671285 
n - 21 

sj  - 879.46935 
S*  - 812.36 

Fc  * Sd/*2  * 1*0826103 
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directly  relevant  to  entropy  and  statistical  test  calcula- 
tions was  applied  separately  in  making  those  calculations. 
The  results  are  reflected  in  Table  21  of  Appendix  D.  In 
only  one  instance  was  the  null  hypothesis  rejected.  That 
rejection  occurred  at  the  .10  level  of  significance  for 
the  data  representing  Table  3,  "Entropy  Data  Collection 
Results,"  or  the  calculated  arithmetic  means  of  the  aspect 
feature  probability  outcomes  (16:50).  Thus,  the  authors  of 
the  current  study  believe  that,  regardless  of  the  apparent 
typographical  errors  in  the  data  as  presented  in  the 
Glover-Lenz  study,  Glover  and  Lenz  were  probably  correct 
in  their  finding  of  nonrejection  of  the  null  and,  thereby, 
the  research  hypothesis  based  upon  their  method  of  statis- 
tical testing. 


APPLICABILITY  OF  THE 
STATISTICAL  TEST 


Aside  from  any  possible  doubts  as  to  the  adequacy 
of  the  sample  sizes  and  the  assumptions  of  normality,  the 
assumed,  independence  of  the  variable  time  (t^) , error  (e^ , 
deviation  (d^) , and  the  actual  observed  cost  (ci)  from 
previous  outcomes  as  reflected  by  prior  dependent  variables 

x 


or  cost  observations, 


1,  can  be  called  into 


question  in  both  studies  (35:247-257,282,461-463,469-474, 
485-493;  33:481-491). 
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Both  within  the  Glover-Lenz  study  (16:36,65)  and  in 
this  appendix,  the  validity  of  the  assumption  of  linear  cost 
has  been  addressed.  However,  in  regards  to  the  relation- 
ship of  this  assumption  to  the  appropriateness  of  the 
statistical  test  used,  Glover  and  Lenz  go  on  to  state  the 
following  (16:65) : 

The  assumption  of  linear  cost  relative  to  time  may 
be  criticized  as  unrealistic,  and  perhaps  as  invali- 
dating any  results  due  to  possible  inter cor relation  of 
cost  observations  with  respect  to  subsequent  time 
points.  However,  the  random  variables  of  the  statis- 
tical test  were  the  errors  of  the  actual  cost  from  the 
least-squares  fitted  line  and  the  deviations  from  the 
projected  straight  line  from  the  origin  to  the  Martin 
Cost  Model  estimate  of  final  cost.  As  3uch,  the  test 
takes  the  form  of  assessing  whether  the  two  projections 
could  have  come  from  the  same  family  of  possible  linear 
cost  estimates  for  the  SRAM  program.  Such  a test  is 
internally  consistent  with  the  assumptions  of  the  study. 
Note  that  linearity  of  cost  appears  to  be  a typical 
assumption  in  capital  budgeting.  Weapon  system  acqui- 
sition decisions  are  conceptually  similar  to  the 
capital  budgeting  decision. 

Based  upon  a further  review  of  the  literature,  the 
authors  of  the  current  study  have  come  to  believe  that  the 
assumption  of  independence  for  the  above  cited  variables, 
as  implied  by  both  the  statistical  testing  method  employed 
and  the  above  statement  from  the  Glover-Lenz  study,  is 
invalid.  Since  the  dependent  variable  is  the  cumulative 
x 

cost  or  J c^  for  a given  time  point  t^,  where  i = 1, 

i«l 

. . .,  x,  then  both  a trend  of  increasing  cost  and  random- 
tracking patterns  should  occur  over  time.  Thus,  at  least 
under  the  least-squares  method  of  fitting  a line  through 
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the  observed  cost  data,  the  necessary  assumptions  of  inde- 
pendence under  simple,  linear  regression  techniques  is 
inappropriate  for  these  two  studies  (35:268,282).  Further, 

from  a logical  standpoint  it  would  also  appear  that  each 

2 2 

d^  should  be  independent  if  the  variances  (Sfi  and  S^) 
based  upon  e^  and  d^  are  to  be  compared  on  an  equal  footing. 
Similar  to  the  case  of  e^,  each  d^  does  not  appear  to  be 

x 


independent  since  d^  * c^  - where  c^ 

i 


and 


x 

t 

i“l 

fore,  it  appears  that  some  form  of  time  series  analysis 
rather  than  simple  linear  regression  techniques  should 
have  been  employed  so  that  the  relationship  between  the 
variables  and  the  various  patterns  of  dependency  could  be 
differentiated  for  purposes  of  analysis  and  testing. 

• The  appropriateness  of  using  the  actual,  individual 
cumulative  cost  values  as  the  basis  upon  which  to  determine 
the  accuracy  of  the  predicted  final  cost  can  also  be  called 
into  question  from  the  standpoint  of  the  basic  purpose  of 
these  studies.  These  studies  were  designed  to  analyze  and 
test  the  ability  of  the  model  to  predict,  with  accuracy, 
the  final  cost  of  a development  program.  Therefore,  the 
emphasis  should  be  upon  one  ^int  in  time,  T^.  The  values 
of  Cj  and  IE  that  are  used  to  make  that  prediction  originated 


L 


at 


fci 


x (see  Appendix  C,  Table  14) . There- 


mm 





at  time  TQ.  If  the  model  is  to  be  used  in  actually 
identifying  and  in  controlling  potential  and  actual  areas 
of  cost  growth,  then,  as  described  in  Chapter  II,  the  pre- 
diction would  be  updated  as  the  program  progresses  based 
upon  the  measurement  of  IE  and  the  incorporation  of  any 
changes  in  Cj  at  time  period  Tj_.  In  the  case  of  the 
current  study,  which  reflects  somewhat  the  same  situation 
as  depicted  in  Figure  9,  Chapter  II,  the  predicted  cost  of 
the  program  from  the  origin,  TQ,  would  immediately  identify 
the  program  as  being  potentially  out  of  control.  Such 
identification  would  have  resulted  in  a reassessment  of 
the  prediction  and  an  upgrading  of  IE  and/or  Cj  through  the 
subsequent  research  efforts  to  determine  potential  areas 
of  cost  growth  and/or  alternative  outcomes  not  before 
considered. 


hf 


j 1 ^s.-1  ■;%&■ 


APPENDIX  E 

ENTROPY  OUTPUT  DATA 
COMPARISON 

Some  consideration  was  given  to  a close  examination 
and  comparison  of  the  participant  probability  assignment. 

In  Chapter  II  and  Appendix  B,  it  was  explained  that 
participant  4 was  removed  from  the  study  for  specific 
reasons.  Though  removed , the  areas  of  uncertainty  partici- 
pant 4 contributed  were  retained  under  the  assumption  they 
would  not  have  an  adverse  affect  on  the  study.  In  comparing 
the  output  in  Table  23  with  and  without  the  data  provided 
by  participant  number  4,  we  find  little  overall  difference 
in  the  value  calculated  for  relative  entropy.  In  fact, 
when  the  aspects  of  uncertainty  contributed  by  participant 
number  4 are  removed,  the  value  for  total  relative  entropy 
increased  by  only  .011  from  .910  (with)  to  .921  (without). 

In  assigning  probability  valu-.  to  the  DELPHI 
procedure  questions,  there  were  some  completely  opposite 
responses  between  two  participants.  Table  23  represents 
an  attempt  at  combining  the  responses  of  the  participants 
to  reduce  the  "extreme  effect"  of  probability  assignment. 
The  union  of  the  responses  of  participant  number  1 and 
participant  number  3 reduce  this  "effect"  and  contribute  a 
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Table  23 

Entropy  Data  Comparison 


Participants 


1 & 2 
2(3 
1 & 3 
1/  2 « 3 


Relative  Entropy 

A* 

B2 

902 

.949 

.919 

.925 

,726 

.732 

.910 

.921 

? . 


Al 

2 


relative  entropy  values  to  include  areas  of 
uncertainty  from  participant  number  4 


B 


relative  entropy  values  not  to  include  areas  of 
uncertainty  from  participant  number  4 


-I 


lower,  more  desirable  relative  entropy  value  and,  in  turn, 
a higher  value  for  informational  efficacy. 

The  initial  final  development  program  cost 
estimate,  CI#  was  $83,635  million  and  the  actual  final 
cost  estimate,  CA,  was  $89,322  million.  Using  the  Martin 
Cost  Model  calculations,  substituting  CA  for  C£,  and  a 
of  $83,635  million,  a relative  entropy  value  of  .06366 
would  be  required  to  produce  a final  development  cost  of 
$89,322  million.  To  produce  such  results  would  require 
a probability  assignment  of  approximately  .00749  for  one 
of  the  outcomes  of  each  of  the  12  aspects  in  the  current 
study. 
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